2587 Commits

Author	SHA1	Message	Date
Nikolay	5b235b902b	Adds ready probe (#874) ... * adds leading forward slash check for scrapeURL path https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835 * adds ready probe for scrape config initialization, it should prevent metrics loss during vmagent rolling update, /ready api will return 425 http code, if some scrape config still waits for initialization. * updates docs * Update app/vmagent/README.md * renames var * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 20:33:48 +02:00
Aliaksandr Valialkin	2cd86d0220	lib/promscrape: docs update after e4182dd896 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 17:13:34 +02:00
Nikolay	d0a9b24c5a	reduces memory usage for vmagent, (#880) ... * reduces memory usage for vmagent, limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 * Update app/vmagent/README.md * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 17:13:33 +02:00
Aliaksandr Valialkin	c736339843	lib/{storage,mergeset}: clean cached index blocks and inmemory blocks more aggressively ... Previously such blocks were cleaned after they weren't accessed during 10 minutes. Now they are cleaned after one minute of missing access. This should reduce memory usage in general case.	2020-11-04 16:44:15 +02:00
Aliaksandr Valialkin	f35aafb6a5	Revert "lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets" ... This reverts commit b08c6f5144.	2020-11-04 11:45:38 +02:00
Aliaksandr Valialkin	b08c6f5144	lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 11:09:05 +02:00
Aliaksandr Valialkin	c046735571	lib/promscrape: reduce memory allocations in promLabelsString() function ... This should help with reducing memory usage in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 10:38:59 +02:00
Aliaksandr Valialkin	c0bd208c77	lib/storage: do not report about the need of free disk space if parts cannot be merged due to too big write amplification	2020-11-03 15:32:09 +02:00
Aliaksandr Valialkin	1b9778a756	lib/storage: remove unneeded fmt.Sprintf	2020-11-03 14:21:04 +02:00
John Belmonte	8653e2658e	add short_version label to vm_app_version metric (#877) ... * add short_version label to vm_app_version metric use case: Version panel of Grafana dashboard should use a live query, but currently it uses a template query which becomes stale. Grafana is not able to preform regex substitution on labels. * Update metrics.go * fix compile	2020-11-03 14:12:42 +02:00
Aliaksandr Valialkin	f3a7e6f6e3	lib/storage: remove obsolete code	2020-11-02 19:17:30 +02:00
Aliaksandr Valialkin	3ed9f1d5a9	lib/promscrape: properly handle response body after 301 redirect ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/869	2020-11-02 01:09:59 +02:00
Nikolay	e8fe618bbb	fixes panic at scrape error body formating, (#868) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/864 regression after body reuse improvements	2020-11-01 23:25:23 +02:00
Nikolay	058f49de57	adds leading forward slash check for scrapeURL path (#855) ... * fixes in-consistency with prometheus behaviour for scrape targets url path. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835	2020-11-01 23:22:16 +02:00
Aliaksandr Valialkin	ed724d25ba	lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics	2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin	901514be88	lib/storage: drop more samples outside the given retention during background merge ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-10-31 20:44:47 +02:00
Aliaksandr Valialkin	abdf22e0bb	app/vmagent: expose /api/v1/targets page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets ... This page is exposed by vmagent and by a single-node VictoriaMetrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643	2020-10-20 21:55:14 +03:00
Aliaksandr Valialkin	7599e5c835	lib/storage: properly handle the case when key="__name__" is passed to MetricName.AddTag*	2020-10-20 20:09:52 +03:00
Aliaksandr Valialkin	9c5cd5a6c5	lib/storage: code cleanup after 5bfd4e6218	2020-10-20 16:10:53 +03:00
Aliaksandr Valialkin	0db7c2b500	app/vmstorage: support for -retentionPeriod smaller than one month ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-10-20 14:42:46 +03:00
Aliaksandr Valialkin	5518b11720	lib/memory: do not print trailing zeroes in logs for -memory.allowedPercent command-line flag	2020-10-20 14:42:37 +03:00
faceair	0093ee3cd9	disable response compression on websocket (#841)	2020-10-17 13:33:37 +03:00
Aliaksandr Valialkin	efb1989193	lib/storage: small code adjustements after d2960a20e0 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781	2020-10-17 01:17:12 +03:00
faceair	8ddf089deb	evaluate the execution cost of all tag filters (#824) ... * evaluate the execution cost of all tag filters * fix suffixes typo	2020-10-17 01:13:20 +03:00
Nikolay Khramchikhin	a5d842caf8	fixes openstack api endpoint with suffix trim adds openstack (#840) ... api v2.0 check Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728	2020-10-16 23:01:45 +03:00
Aliaksandr Valialkin	938b3b7ed1	lib/promscrape: code prettifying after 9bd9f67718	2020-10-12 16:13:59 +03:00
Nikolay Khramchikhin	7f96712b38	Adds dockerswarm sd (#818) ... * adds dockerswarm service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656 Following roles supported: services, tasks and nodes. Basic, token and tls auth supported. Added tests for labels generation. * added unix socket support to discovery utils Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-12 16:13:58 +03:00
Aliaksandr Valialkin	de1c07b937	lib/backup: add MustStop() method for all remote filesystems	2020-10-09 15:32:13 +03:00
Aliaksandr Valialkin	bf6d523bef	lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter	2020-10-09 15:11:55 +03:00
Aliaksandr Valialkin	d2e917d1cb	app/vmstorage: add vm_rows_added_to_storage_total metric, which shows the total number of rows added to storage since app start	2020-10-09 13:36:17 +03:00
Aliaksandr Valialkin	a93c62cd60	lib/promscrape: fix tests after 71ea4935de	2020-10-08 19:32:48 +03:00
Aliaksandr Valialkin	0d44e371f3	lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors command-line flag in order to suppress duplicate scrape target errors ... Show also original labels for duplicate targets in error message in order to simplify debugging the issue. Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target. This may simplify debugging for target relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651	2020-10-08 18:59:25 +03:00
Aliaksandr Valialkin	ad0950f630	lib/backup/actions: improve logging to be more clear to humans	2020-10-08 14:23:00 +03:00
Aliaksandr Valialkin	b51fa16177	app/vmstorage: add -finalMergeDelay command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it	2020-10-07 17:42:31 +03:00
Aliaksandr Valialkin	54ff78c6c9	lib/protoparser/graphite: support parsing floating-point timestamp like Graphite does ... Such timestamps are rounded to seconds like Carbon does. See b0ba62a62d/lib/carbon/protocols.py (L197)	2020-10-06 11:38:35 +03:00
Aliaksandr Valialkin	1da41177a8	lib/promscrape/discovery/openstack: show expiration time for refreshed OpenStack token in seconds - this is easier to interpret by human	2020-10-06 11:20:36 +03:00
Aliaksandr Valialkin	97b836a6f4	lib/fs: fix GOOS=openbsd build by adding fadviseSequentialRead implementation. ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785	2020-10-05 23:32:28 +03:00
Aliaksandr Valialkin	50316070d6	lib/promscrape/discovery/openstack: code prettifying after cbe3cf683b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728	2020-10-05 18:12:31 +03:00
Nikolay Khramchikhin	b4c77fc6d2	Adds openstack sd (#811) ... * adds openstack service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728 implemented hypervisors and instance discovery with openstack v3 api. Added tests for labeling and data parsing. Added token refresh. * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-05 16:48:54 +03:00
Aliaksandr Valialkin	7505e4f390	lib/promrelabel: make a copy of label with new name for action: labelmap in the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/812	2020-10-05 16:20:22 +03:00
Aliaksandr Valialkin	9a7c863bd8	lib/protoparser/influx: add -influx.maxLineSize command-line flag for configuring the maximum size for a single Influx line during parsing ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/807	2020-10-05 15:19:11 +03:00
Aliaksandr Valialkin	11ddb79aeb	lib/decimal: add tests for negative values passed to maxUpExponent	2020-10-05 14:57:03 +03:00
Aliaksandr Valialkin	d448e07546	lib/decimal: properly calibrate scale for blocks with Inf values ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/805	2020-10-05 14:52:59 +03:00
Aliaksandr Valialkin	fd7dd5064a	lib/storage: code cleanup after 10f2eedee0 ... Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search, since this code became unused after implementing per-day inverted index almost a year ago. While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names such as `foo.bar.baz`).	2020-10-01 19:12:04 +03:00
Aliaksandr Valialkin	3ad7566a87	lib/storage: imrpove cache effectiveness for time series ids matching the given filters ... Previously the maximum cache lifetime has been limited by 10 seconds. Now it is extended up to a day. This should reduce CPU usage in the following cases: * when querying recently added data with small churn rate for time series * when querying historical data	2020-10-01 14:39:46 +03:00
Aliaksandr Valialkin	7c2e4e267a	lib/storage: allow set values higher than 1 for vm_merge_need_free_disk_space if there are multiple partitions with deferred merges due to disk space shortage	2020-09-29 22:53:34 +03:00
Aliaksandr Valialkin	097a4c10dd	app/vmstorage: add metrics for determining whether background merges need additional disk space to complete ... These metrics are: * vm_small_merge_need_free_disk_space * vm_big_merge_need_free_disk_space Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-29 21:47:47 +03:00
Aliaksandr Valialkin	5dca7bbe85	app/vmagent/remotewrite: do not show -remoteWrite.url in logs if -remoteWrite.showURL isn't set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-29 19:49:19 +03:00
Aliaksandr Valialkin	5e998e597a	lib/cgroup: do not adjust the number of detected CPU cores via /sys/devices/system/cpu/online ... The adjustement increases the resulting GOMAXPROC by 1, which looks confusing to users as outlined at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-698595309	2020-09-29 13:55:53 +03:00
Aliaksandr Valialkin	338a53ccf9	lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm	2020-09-29 13:10:37 +03:00
Aliaksandr Valialkin	ef416c72c2	lib/storage: fix 32-bit builds for GOARH=386 or GOARCH=arm	2020-09-29 12:42:25 +03:00
Aliaksandr Valialkin	0b0259c42c	lib/protoparser/prometheus: sort rows before comparing them in TestParseStream, since the order for callback calls is non-deterministic	2020-09-29 12:29:50 +03:00
Aliaksandr Valialkin	8f25206d8c	lib/protoparser/prometheus: fix TestParseStream after 124f78857b	2020-09-29 12:12:33 +03:00
Aliaksandr Valialkin	81cdf2fa14	lib/{fs,filestream}: small consistency-related updates after cc90a548b1	2020-09-29 00:43:20 +03:00
Nikolay Khramchikhin	658a05ef0f	added openbsd implementations (#790) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785 removed fadvise for openbsd, added freespace implemenation for openbsd	2020-09-29 00:43:19 +03:00
Aliaksandr Valialkin	1481d6d8ff	lib/protoparser: avoid copying of buffer read from the network to unmarshal buffer	2020-09-28 17:19:04 +03:00
Aliaksandr Valialkin	8df33bd5c1	app/{vminsert,vmagent}: improve data ingestion speed over a single connection ... Process data obtianed from a single connection on all the available CPU cores.	2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin	7072db75cb	lib/protoparser: use 64KB read buffer instead of default 4KB buffer provided by net/http.Server ... This should reduce syscall overhead when reading big amounts of data	2020-09-28 02:07:19 +03:00
Aliaksandr Valialkin	6d8c23fdbd	app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native ... This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics	2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin	aadbd014ff	all: add native format for data export and import ... The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.	2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin	00ec2b7189	lib/protoparser: use all the available CPU cores for processing ingested data from a single /api/v1/import stream ... Previously a single data ingestion stream to /api/v1/import could load only a single CPU core.	2020-09-26 04:22:06 +03:00
Aliaksandr Valialkin	533bf76a12	lib/storage: correctly use maxBlockSize in various checks ... Previously `maxBlockSize` has been multiplied by 8 in certain checks. This is unnecessary.	2020-09-24 18:13:15 +03:00
Aliaksandr Valialkin	543f3aea97	all: consistently use "%w" formatting in fmt.Errorf for wrapped errors	2020-09-23 22:48:21 +03:00
Aliaksandr Valialkin	90d2549428	lib/persistentqueue: protect from multiple concurrent opening for the same persistent queue	2020-09-23 02:17:53 +03:00
Aliaksandr Valialkin	89d652b583	lib/cgroup: attempt to obtain available CPU cores via /sys/devices/system/cpu/online ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-674423728	2020-09-22 23:27:26 +03:00
Aliaksandr Valialkin	31e341371b	lib/storage: code prettifying after be5e1222f3 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781	2020-09-22 00:42:20 +03:00
faceair	ad41e39350	add filter to getMetricIDs (#783) ... * add getMetricIDs filter * check nil filter before use	2020-09-22 00:42:19 +03:00
Aliaksandr Valialkin	d32c3f747c	lib/logger: add -loggerDisableTimestamps command-line flag for disabling timestamps in logs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/778	2020-09-21 19:25:50 +03:00
Aliaksandr Valialkin	f961838290	lib/promscrape/discovery/ec2: code prettifying after 312fead9a2	2020-09-21 18:44:04 +03:00
Nikolay Khramchikhin	0069353d5e	Add improvements to ec2_sd_discovery (#775) ... * Add improvements to ec2 discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771 role_arn support with aws sts instance iam_role support refreshing temporary tokens * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * changed implementation, removed tests, clean up code * moved endpoint builder into getEC2APIResponse Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-21 16:05:01 +03:00
Aliaksandr Valialkin	a9321f6a60	lib/storage: reduce CPU load for idle VictoriaMetrics by reducing the frequency for the need for background merges	2020-09-21 15:51:26 +03:00
Aliaksandr Valialkin	604e8f6114	lib/decimal: optimize maxUpExponent() by eliminating division from hot path	2020-09-19 13:49:43 +03:00
Aliaksandr Valialkin	df547bf345	lib/persistentqueue: sync data to file inside filestream.Writer.MustFlush	2020-09-19 12:51:46 +03:00
Aliaksandr Valialkin	778ea183ca	lib/decimal: properly store Inf values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752	2020-09-18 19:08:53 +03:00
Aliaksandr Valialkin	adb4d3b75c	lib/persistentqueue: flush data to disk every second ... Previously small amounts of data may be left unflushed for extended periods of time if vmagent collects small amounts of data. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687	2020-09-18 13:06:03 +03:00
Aliaksandr Valialkin	8fd791d399	lib/promscrape: avoid copying response body when scraping targets. ... This should reduce memory usage when scraping targets with millions of metrics.	2020-09-18 13:06:00 +03:00
Aliaksandr Valialkin	d96858b921	lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions ... This may be useful for freeing up storage space after time series deletion. See https://victoriametrics.github.io/#force-merge for more details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-17 12:20:56 +03:00
Aliaksandr Valialkin	3abbb38254	lib/{mergeset,storage}: compare errors with errors.Is()	2020-09-17 03:03:10 +03:00
Aliaksandr Valialkin	ddb3519e17	lib/{mergeset,storage}: code prettifying	2020-09-17 02:06:37 +03:00
Aliaksandr Valialkin	bf826dd828	lib/storage: removed duplicate checks for empty parts during merge - another check is in the beginning of mergeParts functions	2020-09-17 01:49:08 +03:00
Aliaksandr Valialkin	406f4fe445	app/vmagent: substitute -remoteWrite.url with secret-url value in logs, since it may contain sensitive info such as passwords or auth tokens ... Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-16 22:36:18 +03:00
Aliaksandr Valialkin	9705ac5d7a	lib/persistentqueue: code simplification after d455764a6f	2020-09-16 21:14:01 +03:00
Aliaksandr Valialkin	eee6f1e56d	lib/persistentqueue: make the persistent queue more durable against unclean shutdown (kill -9, OOM, hard reset) ... The strategy is: - Periodical flushing of inmemory blocks to files, so they aren't lost on unclean shutdown. - Periodical syncing of metadata for persisted queues, so the metadata remains in sync with the persisted data. - Automatic adjusting of too big chunk size when opening the queue. The chunk size may be bigger than the writer offset after unclean shutdown. - Skipping of broken chunk file if it cannot be read. - Fsyncing finalized chunk files. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687	2020-09-16 18:13:24 +03:00
Aliaksandr Valialkin	6ce52e3702	lib/protoparser/vmimport: add more testcases for invalid timestamps and values ... Updates https://github.com/VictoriaMetrics/vmctl/issues/25	2020-09-16 02:21:53 +03:00
Aliaksandr Valialkin	cd87ca303f	lib/protoparser: report more errors for incorrect timestamps and/or values ... Previously certain errors in timestamps and/or values could be silently skipped, which could lead to samples with zero values stored in the database. Updates https://github.com/VictoriaMetrics/vmctl/issues/25	2020-09-16 02:16:15 +03:00
Aliaksandr Valialkin	5c4e111b43	lib/protoparser/graphite: return error when value or timestamp cannot be properly parsed ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99	2020-09-16 02:16:13 +03:00
Aliaksandr Valialkin	0c1c1b79ba	lib/promscrape: add a link to troubleshooting docs to error message when duplicate scrape target with identical labels is skipped	2020-09-15 14:16:20 +03:00
Aliaksandr Valialkin	ca08161b54	lib/promscrape: typo fix	2020-09-12 00:14:15 +03:00
Aliaksandr Valialkin	e53235ac5c	lib/promscrape: do not reset the remaining rows when pushing a part of data to remote storage during big scrapes ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/753 Thanks to @PerGon and @clmssz for help with debugging.	2020-09-11 23:38:17 +03:00
Aliaksandr Valialkin	114cf24b43	lib/promscrape/discovery/dns: add __meta_dns_srv_record_target and __meta_dns_srv_record_port labels ... This syncs dns service discovery with Prometheus 2.21 - see https://github.com/prometheus/prometheus/releases and https://github.com/prometheus/prometheus/pull/7678 .	2020-09-11 21:35:39 +03:00
Aliaksandr Valialkin	5cf5a0e8c4	lib/protoparser/common: do not read request body when parsing timestamp query arg ... This was preventing from reading data via /api/v1/prometheus/import . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 14:45:21 +03:00
Aliaksandr Valialkin	81c05f669b	lib/storage: do not store inf values, since they may lead to significant precision loss for previously stored values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752	2020-09-11 14:45:20 +03:00
Aliaksandr Valialkin	8bd5aa3516	lib/protoparser: accept timestamp in milliseconds instead of seconds at /api/v1/import/prometheus ... This improves consistency with timestamps in Prometheus text exposition format	2020-09-11 14:05:24 +03:00
Aliaksandr Valialkin	58d3b82ae5	app/{vminsert,vmagent}: allow passing timestamp via timestamp query arg when ingesting data to /api/v1/import/prometheus ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 13:28:31 +03:00
Nikolay Khramchikhin	af994562c8	Added endpointslices discovery to k8s api (#760) ... This is similar to https://github.com/prometheus/prometheus/pull/6838 , which will be added in Prometheus v2.21. See https://github.com/prometheus/prometheus/releases/tag/v2.21.0-rc.1 * Added endpointslices discovery to k8s api Started from 1.17 k8s version endpointslices is beta, it allows to query k8s api for endpoints more efficient. It presents at scrape_config.yaml as separate role for kubernetes_sd_config. kubernetes_sd_config: - role: endpointslices * fixed typos, changed EndpointConditions signature - with values instead of pointers	2020-09-11 12:24:50 +03:00
Aliaksandr Valialkin	f307e6f432	app/vmselect: initial implementation of Graphite Metrics API ... See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api	2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin	f5cb213ef9	lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps ... This should reduce disk space usage when scraping targets containing metrics with identical names such as `node_cpu_seconds_total`, histograms, quantiles, etc. Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring the effectiveness of timestamp blocks merging.	2020-09-09 23:59:21 +03:00
Aliaksandr Valialkin	e5c8377212	lib/httpserver: add a jitter to connection timeouts in order to protect from Thundering herd problem	2020-09-08 19:57:20 +03:00
Nikolay Khramchikhin	fb356c434b	Changed s3 configProfile flag default, (#749) ... aws sdk has complicated logic for chosing profile name and we shouldn't set it to `default` value. It leads to bugs and improper configuration. Set it to empty value by default is safe. It will be automatically set to `default` by sdk.	2020-09-07 21:55:45 +03:00
Aliaksandr Valialkin	d2f6f96e4a	lib/memory: fall back to reading hierarchical memory limit in cgroups when the default limit isn't set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/699	2020-09-04 00:04:57 +03:00
Aliaksandr Valialkin	281d715060	lib/httpserver: add -http.connTimeout command-line flag for limiting the lifetime for incoming http connections ... This can be useful for balancing incoming connections among multiple services.	2020-09-03 22:23:55 +03:00
Aliaksandr Valialkin	4fa97430d7	app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats ... Extra labels may be added to the imported data by passing `extra_label=name=value` query args. Multiple query args may be passed in order to add multiple extra labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719	2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin	95ce89e7d7	lib/promscrape: use the number of parsed rows as a basis for writeRequestCtxPool leveling ... The previous basis on `cap(sw.labels)` doesn't work anymore after 7785869ccc , because `sw.labels` may be reset multiple times when processing big number of rows.	2020-09-02 18:46:55 +03:00
Aliaksandr Valialkin	bc1ca4b20b	lib/httpserver: add -http.idleConnTimeout command-line flag for tuning the timeout for incoming idle http connections	2020-09-01 15:33:31 +03:00
Aliaksandr Valialkin	a01c56104a	lib/promscrape: fix applying sample_limit when scraping targets with big number of metrics ... This has been broken at 7785869ccc	2020-09-01 11:09:25 +03:00
Aliaksandr Valialkin	deff8d419a	lib/promscrape: reduce memory usage when scraping targets with millions of metrics ... This should help when scraping /federate endpoints from Prometheus instances, which scrape millions of metrics. See https://prometheus.io/docs/prometheus/latest/federation/	2020-09-01 10:55:24 +03:00
Aliaksandr Valialkin	5f2277624a	lib/{promscrape,leveledbytebufferpool}: rename getPoolIdAndCapacity to getPoolIDAndCapacity in order to make golint happy	2020-08-28 09:49:22 +03:00
Aliaksandr Valialkin	45e770ed20	lib/cgroup: limit the maximum GOMAXPROCS value to the number of available CPU cores ... There is no sense in setting GOMAXPROCS to value higher than the number of available CPU cores.	2020-08-28 09:49:22 +03:00
Roman Khavronenko	34ef10fbcc	lib/flagutil: avoid int overflow for arch 386 (#710) ... Arch 386 is a 32-bit architecture and interprets int type for numbers as an explicit int32, whereas on most modern CPUs int is implicitly an int64. This makes tests to fail with `int overflow` error.	2020-08-28 09:46:35 +03:00
Aliaksandr Valialkin	9a77ae9d1c	lib/promscrape: reduce memory usage when scraping targets with big number of metrics alongside targets with small number of labels ... Previously targets with big number of metrics and/or labels could generated too big buffers, which then could be re-used when scraping targets with small number of metrics. This resulted in memory waste. Now big buffers are used only for targets with big number of metrics / labels, while small buffers are used for targets with small number of metrics / labels.	2020-08-16 22:30:34 +03:00
Aliaksandr Valialkin	3ea6444219	lib/leveledbytebufferpool: allocate byte buffers with capacity rounded to the upper boundary for the given bucket ... This should reduce the number of resizings for the returned byte buffers.	2020-08-16 22:13:38 +03:00
Roman Khavronenko	4b89da9463	lib/decimal: rename significant decimal digits to significant figures (#698) ... The previous notion was inconsistent with what `decimal.Round` does. According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding applied to all significant figures, not just decimal ones.	2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin	6aab2f4989	all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin	1b5467f7fd	lib/memory: improve log message about the memory allowed to use by VictoriaMetrics	2020-08-16 17:08:27 +03:00
Aliaksandr Valialkin	d9f7ea1c6e	lib/protoparser: removed unnecessary call to SetReadDeadline when reading a stream of data ... The OS should return any buffered data in the stream without the need to set the read timeout. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-15 15:38:35 +03:00
Aliaksandr Valialkin	edf3ee2a9b	lib: dump compressed block contents on error during decompression ... This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222	2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin	6e863376f7	lib/leveledbytebufferpool: pre-allocate byte slice with the given capacity if the pool is empty ... This should reduce memory allocations and copying when the byte slice is growing.	2020-08-15 01:41:59 +03:00
Aliaksandr Valialkin	3efa4e4e1c	lib/protoparser: move common code for detecting timeouts to ReadLinesBlockExt ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-14 20:39:51 +03:00
Aliaksandr Valialkin	c6b0547847	lib/protoparser: prevent from busy loop on repeated timeout errors when reading streams of ingested data ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-14 20:13:37 +03:00
Aliaksandr Valialkin	9d79a3a99d	lib/memory: add -memory.allowedBytes command-line flag for setting absolute memory limit for VictoriaMetrics caches	2020-08-14 19:19:10 +03:00
Aliaksandr Valialkin	b996280c65	app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag	2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin	c82a485cf6	lib/protoparser/prometheus: typo fix in error message	2020-08-14 11:04:15 +03:00
Aliaksandr Valialkin	9e67343756	lib/promscrape: use a hint on body length instead of body capacity ... This should reduce memory usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689	2020-08-14 01:17:46 +03:00
Aliaksandr Valialkin	b4119bb51e	lib/promscrape: reduce memory usage when scraping big number of targets ... Thanks to @dxtrzhang for the original idea at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/688 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689	2020-08-14 01:05:04 +03:00
Aliaksandr Valialkin	1724cc241e	lib/promscrape: properly retry requests on the server closed connection before returning the first response byte error during service discover API calls and target scrapes	2020-08-13 22:32:29 +03:00
Aliaksandr Valialkin	60c7397be5	all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components ... Such placeholders are substituted by the corresponding environment variable values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583	2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin	6721e47ae9	app: respect CPU limits set via cgroups ... Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage for cases when VictoriaMetrics components run in containers with CPU limits. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin	59d95961b8	lib/protoparser: clarify that the string passed to Unmarshal() function must remain available when the parsed rows are in use	2020-08-11 17:05:21 +03:00
Aliaksandr Valialkin	890cfe5b61	lib/protoparser/influx: accept precision=us and precision=µ according to https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint	2020-08-10 20:23:20 +03:00
Aliaksandr Valialkin	e3439a6cd0	lib/promscrape: optimize per-metric hash calculations ... This increases vmagent performance by up to 10% when scraping big number of metrics	2020-08-10 19:47:50 +03:00
Aliaksandr Valialkin	4ce1368e4b	lib/storage: mention time range used in the query that led to error message ... This should improve detecting slow queries with too big time ranges	2020-08-10 13:46:29 +03:00
Aliaksandr Valialkin	f92255e803	lib/storage: mention tag filters used in the query that led to error message ... This should improve detecting invalid or heavy queries that lead to errors.	2020-08-10 13:36:54 +03:00
Aliaksandr Valialkin	b3d4ff7ee2	app/vmstorage: improve error logging when the request times out	2020-08-10 13:17:24 +03:00
Aliaksandr Valialkin	e3999ac010	lib/promscrape: show real timestamp and real duration for the scape on /targets page ... Previously the scrape duration may be negative when calculated scrape timestamp drifts away from the real scrape timestamp	2020-08-10 12:40:49 +03:00
Aliaksandr Valialkin	c09c881264	lib/promscrape: make errcheck happy	2020-08-09 13:17:30 +03:00
Aliaksandr Valialkin	2dfb42a8b4	lib/promscrape: export scrape_samples_added per-target metric like Prometheus does ... This metric may be useful for detecting targets with high churn rate for the exported metrics. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/683	2020-08-09 12:45:30 +03:00
Aliaksandr Valialkin	fd9f1463df	lib/fs: use WARN instead of ERROR log level for the message when NFS diretory removal temporarily fails ... this is expected condition, so it is better to use WARN log level for it	2020-08-09 12:07:35 +03:00
Aliaksandr Valialkin	d4be3efc60	lib/promscrape: add a test for scrape config for blackbox exporter ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/684	2020-08-09 12:03:51 +03:00
Aliaksandr Valialkin	67cacb22ac	lib/httpserver: add -tls, -tlsCertFile and -tlsKeyFile command-line flags in every vm binary ... This makes such binaries compatible with binaries from `master` branch (aka single-node version) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/677	2020-08-07 10:57:32 +03:00
Aliaksandr Valialkin	307281e922	lib/storage: slow down concurrent searches when the number of concurrent inserts reaches the limit ... This should improve data ingestion performance when heavy searches are executed See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-08-07 08:49:13 +03:00
Aliaksandr Valialkin	dd1d59f57a	lib/storage: properly check timeouts and pace limits ... Previously they were checked on every iteration for small number of iterations	2020-08-07 08:40:56 +03:00
Aliaksandr Valialkin	a2039b3bbc	app/vmselect: return the upper bound on the number of found time series from storage.Search.Init ... This is used by a single-node version in order to reduce memory allocations during search. See bc8381613d for details.	2020-08-06 19:20:31 +03:00
Aliaksandr Valialkin	b690eeff53	lib/storage: reduce the frequency (and overhead) for timeout and pace limiter checks by 4x	2020-08-06 18:45:47 +03:00
Aliaksandr Valialkin	6c0a92a1ee	lib/pacelimiter: increase scalability for multi-CPU system	2020-08-06 18:33:07 +03:00
Aliaksandr Valialkin	13f8644f8e	lib/storage: optimize prefetching metric names for the given metricIDs	2020-08-06 16:52:58 +03:00
Aliaksandr Valialkin	f789e0fa44	lib/fs: export vm_nfs_pending_dirs_to_remove metric for monitoring the number of pending directories that couldn't be removed due to NFS lock	2020-08-06 15:31:50 +03:00
Aliaksandr Valialkin	a3e91c593b	lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 ... This should limit the maximum memory usage and reduce CPU trashing on vmstorage when multiple heavy queries are executed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-08-05 18:27:21 +03:00
Aliaksandr Valialkin	76064ba9e7	Perform conversion from string to []byte according to rule #6 at https://golang.org/pkg/unsafe/#Pointer	2020-08-05 11:55:12 +03:00
Aliaksandr Valialkin	8cc2e01386	lib/backup: allow using ~/.aws/config without region ... Use us-west-2 for determining bucket region.	2020-08-04 13:08:05 +03:00
Aliaksandr Valialkin	a2aa3a60eb	app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin	3149af624d	lib/storage: reduce the maximum number of concurrent merge workers to GOMAXPROCS/2 ... Previously the limit has been raised to GOMAXPROCS, but it has been appeared that this increases query latencies since more CPUs are busy with merges. While at it, substitute `*MergeConcurrencyLimitCh` channels with simple integer limits.	2020-07-31 17:53:13 +03:00
Aliaksandr Valialkin	29bbab0ec9	lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected ... The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts. Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon, which should return CPU and disk IO usage to normal levels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin	96039dcb40	lib/storage: properly update vm_slow_row_inserts_total metric when importing multiple data points per time series at once ... Previously the `vm_slow_row_inserts_total` metric may be incremented multiple times for different data points per a single time series, while only a single increment is needed when inserting the first data point for this time series.	2020-07-30 16:17:19 +03:00
Aliaksandr Valialkin	1e067401ba	lib/httpserver: emit X-Forwarded-For additionally to remoteAddr in error logs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-29 13:12:35 +03:00
Sasasu	96bc476e53	lib/storage: metaindexRow use memroy more efficiently (#655) ... due to memory align the metaindexRow structure use 64-byte pre object. this commit changes the order of field, make metaindexRow use 56-byte pre object. Signed-off-by: Sasasu <su@sasasu.me>	2020-07-27 23:23:25 +03:00
Aliaksandr Valialkin	f26ef58137	lib/protoparser/prometheus: add a test for cassandra-exporter ... Thanks to Seva	2020-07-27 18:37:46 +03:00
Aliaksandr Valialkin	94cc677b0c	lib/storage: slightly reduce code difference between single-node and cluster versions	2020-07-24 01:18:05 +03:00
Aliaksandr Valialkin	fb3d1380ac	lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index ... Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`. This commit stops searching in inverted index on query timeout.	2020-07-23 21:22:05 +03:00
Aliaksandr Valialkin	dbf3038637	lib/storage: add more fine-grained pace limiting for search	2020-07-23 19:21:49 +03:00
Aliaksandr Valialkin	b8303afcd8	lib/storage: improve prioritizing of data ingestion over querying ... Prioritize also small merges over big merges. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin	7d0743422b	lib/storage: properly calculate global metrics in UpdateStats()	2020-07-23 00:35:31 +03:00
Aliaksandr Valialkin	6afdcf8a20	lib/mergeset: properly calculate global metrics in UpdateStats() ... Previously these metrics could be calculated multiple times for multiple mergeset.Table instances.	2020-07-23 00:35:29 +03:00
Aliaksandr Valialkin	23fa44e56e	lib/storage: reorder mergeBlockStreams() args in order to make them more consistent	2020-07-22 21:58:25 +03:00
Aliaksandr Valialkin	754eac676d	lib/storage: prevent possible race condition when all the goroutines exit Storage.AddRows, before goroutines other goroutines are blocked on searchTSIDsCond inside Storage.searchTSIDs ... This condition may occur after the following sequence of events: 1) A goroutine enters the loop body when len(addRowsConcurrencyCh) == cap(addRowsConcurrencyCh) inside Storage.searchTSIDs. 2) All the goroutines return from Storage.AddRows. 3) The goroutine from step 1 blocks on searchTSIDsCond.Wait() inside the loop body. The goroutine remains blocked until the next call to Storage.AddRows, which calls searchTSIDsCond.Signal(). This may take indefinite time.	2020-07-22 21:52:42 +03:00
Aliaksandr Valialkin	a3f48e395e	app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage	2020-07-21 21:55:42 +03:00
Aliaksandr Valialkin	67be79a0bc	lib/uint64set: optimize adding items to the set via Set.AddMulti	2020-07-21 20:57:05 +03:00
Aliaksandr Valialkin	31ef39e8da	lib/httpserver: log remote address in error message from httpserver.Errorf ... This should improve detection of the root cause of errors. Thanks to Anant for the idea.	2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin	be0ab4fbfe	lib/storage: reset MetricName->TSID cache after marking metricIDs as deleted ... This is a follow-up commit after 12b16077c4 , which didn't reset the `tsidCache` in all the required places. This could result in indefinite errors like: missing metricName by metricID ...; this could be the case after unclean shutdown; deleting the metricID, so it could be re-created next time Fix this by resetting the cache inside deleteMetricIDs function.	2020-07-14 14:05:19 +03:00
Aliaksandr Valialkin	a4c96d9e6d	lib/protoparser: properly update vm_protoparser_rows_read_total{type="promscrape"} metric	2020-07-14 12:15:56 +03:00
Seva Poliakov	a5e713b6e0	add vm_protoparser_rows_read_total metrics to promscrape (#624) ... * add vm_protoparser_rows_read_total metrics to promscrape move vm_protoparser_rows_read_total for promscrape to better place move vm_protoparser_rows_read_total for promscrape to better place * remove possibility of infinity loop at prometheus parser	2020-07-14 12:02:25 +03:00
Roman Khavronenko	207e93b50d	lib/flagutil: specify additional description for all Array type flags (#620) ... Array type flag is now defined as `value` type in flag description when printed. This change adds additional description to every Array type flag so it would be clear what exact type is used: ``` -remoteWrite.urlRelabelConfig array Optional path to relabel config for the corresponding -remoteWrite.url Supports array of values separated by comma or specified via multiple flags. ```	2020-07-13 22:00:03 +03:00
Roman Khavronenko	605711bde5	lib/persistentqueue: add vm_persistentqueue_bytes_pending metric (#619) ... Metric `vm_persistentqueue_bytes_pending` is a gauge that shows current amount of bytes in persistentqueue flushed on disk as a difference between write and read offsets. This metric is very similar to `vmagent_remotewrite_pending_data_bytes` except of accounting for bytes in-memory.	2020-07-13 21:54:54 +03:00
Roman Khavronenko	a02097e657	Extend metric vm_promscrape_targets with status label (#615) ... The change to `vm_promscrape_targets` metric suppose to improve observability for `vmagent` so it will be possible to track how many targets are up or down for every specific scrape group: ``` vm_promscrape_targets{type="static_configs", status="down"} 1 vm_promscrape_targets{type="static_configs", status="up"} 2 ```	2020-07-13 21:54:53 +03:00
Aliaksandr Valialkin	6373d377ef	app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus	2020-07-10 12:13:28 +03:00
Aliaksandr Valialkin	2012e294d1	properly calculate readCalls	2020-07-10 12:01:05 +03:00
Aliaksandr Valialkin	87f8c728bf	lib/promscrape: send Accept header similar to Prometheus when scraping targets ... This should fix scraping Spring Boot servers, which return incorrect response unless `Accept: text/plain` request header is set. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/608	2020-07-08 19:50:06 +03:00
Aliaksandr Valialkin	7335743d57	lib/storage: limit the maximum concurrency for data ingestion to GOMAXPROCS ... Previously the concurrency has been limited to GOMAXPROCS*2. This had little sense, since every call to Storage.AddRows is bound to CPU, so the maximum ingestion bandwidth is achieved when the number of concurrent calls to Storage.AddRows is limited to the number of CPUs, i.e. to GOMAXPROCS.	2020-07-08 17:34:27 +03:00
Roman Khavronenko	929ad74de6	lib/protoparser: fix metric name of unmarshal errors in promremotewrite (#607) ... The change fixes the typo in metric name `vm_protoparser_unmarshal_errors` to respect the naming standard.	2020-07-08 14:19:27 +03:00
Aliaksandr Valialkin	e401b8d527	lib/protoparser/graphite: go fmt	2020-07-08 14:13:06 +03:00
Aliaksandr Valialkin	50ecf09042	lib/protoparser/graphite: add more tests after eb45185eef ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/610	2020-07-08 14:13:03 +03:00
Seva Poliakov	1ae0334e17	Fix graphite minus one timestamp (#609) ... * fix graphite -1 timestamp * format the graphite fix -1 timestamp	2020-07-08 14:13:01 +03:00
Aliaksandr Valialkin	fad008df7e	lib/storage: clarify out of retention period error message by mentioning -retentionPeriod command-line flag	2020-07-08 13:54:13 +03:00
Aliaksandr Valialkin	fe58462bef	lib/storage: reset MetricName->TSID cache after deleting time series ... This should prevent from adding new data points to deleted time series without the need to check for the deleted time series. This improves ingestion performance a bit when the `deleted time series ids` aka `dmis` set contains big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/596 Based on the idea from @n4mine at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/604	2020-07-06 22:01:24 +03:00
Aliaksandr Valialkin	77bb0e6595	lib/fs: clarify description for -fs.disableMmap command-line flag	2020-07-06 14:28:57 +03:00
Aliaksandr Valialkin	0bff96fe4b	lib/storage: prioritize data ingestion over heavy queries ... Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream. Prevent this by delaying queries' execution until free resources are available for data ingestion. Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources for data ingestion and/or for executing heavy queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291	2020-07-05 19:44:04 +03:00
Aliaksandr Valialkin	6f1d926698	lib/promscrape: use HostClient.DoDeadline instead of HostClient.Do in order to guarantee strict deadline across multiple scrape attempts	2020-07-03 21:33:48 +03:00
Aliaksandr Valialkin	ee03b4ccbd	lib/promscrape: prevent from too big deadline misses on scrape retries ... The maximum deadline miss duration is reduced to 2x scrape_interval in the worst case. By default it is limited to scrape_interval configured for the given scrape target.	2020-07-03 20:42:09 +03:00
Aliaksandr Valialkin	dfa83a4a35	lib/promscrape: check for nil error before checking for the returned status code when scraping targets	2020-07-03 18:37:25 +03:00
Aliaksandr Valialkin	8bb3622e9d	app/vminsert: prevent from adding and/or selecting labels with empty values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600	2020-07-02 23:17:12 +03:00
Aliaksandr Valialkin	6ebac3ab63	app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries ... See https://victoriametrics.github.io/#relabeling	2020-07-02 20:36:33 +03:00
Aliaksandr Valialkin	2361ad8ab4	lib/promscrape: add ability to set disable_compression and disable_keepalive options in scrape_config section of the config passed to -promscrape.config ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-02 14:19:34 +03:00
Aliaksandr Valialkin	0f754bea49	lib/promscrape: add -promscrape.disableKeepAlive command-line flag for disabling http keep-alive connections when scraping targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-01 02:20:46 +03:00
Aliaksandr Valialkin	4cb3e7595c	app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention	2020-07-01 00:58:42 +03:00
Aliaksandr Valialkin	81e3d4305f	lib/httpserver: add Unwrap method to ErrorWithStatusCode, so As and Is functions in standard errors package may properly unwrap the error inside ErrorWithStatusCode	2020-07-01 00:53:49 +03:00
Aliaksandr Valialkin	fe77d661b3	all: use errors.As instead of type assertion for detecting net.Error	2020-07-01 00:16:13 +03:00
Aliaksandr Valialkin	0c4e8aeb2b	all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode	2020-07-01 00:03:11 +03:00
Aliaksandr Valialkin	d962568e93	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin	5a43842bd3	lib/promscrape: add missing label sorting for autogenerated metrics ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/592	2020-06-29 22:39:40 +03:00
Ween	b42cf33c4d	Fix Auto metrics relabeled errors (#593) ... * Fix Auto metrics relabeled errors * Finalize auto-genenated Labels * Fix Test Errors Co-authored-by: xinyulong <xinyulong@kuaishou.com>	2020-06-29 22:39:39 +03:00
Aliaksandr Valialkin	aad38c8283	lib/promrelabel: properly apply ^ and $ anchors to regex value in Prometheus relabeling rules	2020-06-25 17:19:02 +03:00
Aliaksandr Valialkin	fd7a3d880e	lib/fs: go fmt	2020-06-23 23:03:08 +03:00
Aliaksandr Valialkin	08edb90814	lib/fs: fall back to cgo copy for copying the last 4KB of mmaped data ... This probably should fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581	2020-06-23 22:55:56 +03:00
Aliaksandr Valialkin	3a444bb7bb	lib/promrelabel: add support for keep_if_equal and drop_if_equal actions to relabel configs ... These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values. For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port equals __meta_kubernetes_pod_container_port_number: - action: keep_if_equal source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]	2020-06-23 17:29:19 +03:00
Aliaksandr Valialkin	de7e585ac8	lib/promscrape: preserve the previously discovered targets on discovery errors per each job_name ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/582	2020-06-23 15:42:46 +03:00
Aliaksandr Valialkin	521c657f8d	lib/fs: an attempt to fix SIGBUS error by rounding mmap`ed region to multiple of 4KB pages ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581	2020-06-23 13:40:20 +03:00
Aliaksandr Valialkin	5fb60dd647	lib/logger: add -loggerErrorsPerSecondLimit for limiting the rate of ERROR messages	2020-06-23 12:42:59 +03:00
Aliaksandr Valialkin	a80e852aab	lib/promscrape: retry performing the request to the server for up to 3 times before giving up when it closes keep-alive connections ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-06-23 12:34:12 +03:00
Aliaksandr Valialkin	70bf8218bb	app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577	2020-06-21 16:32:27 +03:00
Aliaksandr Valialkin	50aa34bcbe	lib/promscrape/discovery/consul: reduce load on Consul when discovering big number of targets by using background caching ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-06-20 18:20:07 +03:00
Aliaksandr Valialkin	62e1908986	lib/promscrape: reduce default value for -promscrape.discovery.concurrency from 500 to 100 ... This should reduce load on Kubernetes API server and Consul when big number of targets are discovered Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-06-20 17:53:48 +03:00
Aliaksandr Valialkin	1f2826bae2	lib/promscrape/discovery/ec2: expose __meta_ec2_ami like the next Prometheus release will do ... See b5d61fb66c for details	2020-06-20 17:45:30 +03:00
Tristan Su	c254b683fd	lib/storage: set big/small merge concurrency (#568) ... fixed #567 Co-authored-by: Tristan Su <suqing.sq@alibaba-inc.com>	2020-06-19 02:21:55 +03:00
Aliaksandr Valialkin	2e5b6220a4	lib/promrelabel: allows regex capture groups in target_label like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/569	2020-06-19 02:20:58 +03:00
Aliaksandr Valialkin	4f673a5201	app/vminsert: export metrics for determining ingested rows with dropped or truncated labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565	2020-06-19 01:12:44 +03:00
Aliaksandr Valialkin	5f3a895c23	lib/storage: add key!=".+" filter additionally to negative filter matching empty value such as key!~"|foo" ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546	2020-06-18 20:05:45 +03:00
Aliaksandr Valialkin	c40f29f783	lib/storage: properly match {tag!="|foo"} filters ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546	2020-06-10 19:34:37 +03:00
Aliaksandr Valialkin	9f55dea162	lib/httpserver: do not flush and do not close gzip writer if response compression is disabled ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535	2020-06-05 21:37:46 +03:00
Aliaksandr Valialkin	cf91a94daf	lib/backup: properly create missing parent directories in fs.CreateFile	2020-06-05 19:28:25 +03:00
Aliaksandr Valialkin	ba1f764b29	lib/fs: optimize queries that read recent samples for big number of time series ... Use standard copy() func instead of mmap-aware copy func for reading recently touched mmap-ed data. This improves read performance by up to 4x.	2020-06-05 19:10:22 +03:00
Aliaksandr Valialkin	2358d9e41d	lib/fs: add a benchmark for ReaderAt.MustReadAt	2020-06-05 19:10:21 +03:00
Aliaksandr Valialkin	8b0d9df51d	lib/backup/fsremote: create all the parent directories before creating file in CreateFile	2020-06-05 10:25:28 +03:00
Aliaksandr Valialkin	3d0a0b3785	lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds	2020-06-04 13:14:04 +03:00
Vyacheslav Mitrofanov	89a922fb19	allow to use values lower than 10 with the flag -memory.allowedPercent (#531) ... Co-authored-by: Vyacheslav Mitrofanov <vmitrofanov@mfms.ru>	2020-06-03 23:40:13 +03:00
Aliaksandr Valialkin	304f9499cf	lib/bytesutil: prevent from garbage collecting s before returning from ToUnsafeBytes	2020-06-03 00:23:27 +03:00
Aliaksandr Valialkin	eca1afdc20	lib/storage: fix Graphite wildcard matching, which has been broken in v1.36.0	2020-05-28 11:58:47 +03:00
Aliaksandr Valialkin	b0131c79b6	lib/storage: improve search speed for time series matching Graphite whildcards such as foo.*.bar.baz ... Add index for reverse Graphite-like metric names with dots. Use this index during search for filters like `__name__=~"foo\\.[^.]*\\.bar\\.baz"` which end with non-empty suffix with dots, i.e. `.bar.baz` in this case. This change may "hide" historical time series during queries. The workaround is to add `[.]*` to the end of regexp label filter, i.e. "foo\\.[^.]*\\.bar\\.baz" should be substituted with "foo\\.[^.]*\\.bar\\.baz[.]*".	2020-05-27 21:48:08 +03:00
Aliaksandr Valialkin	301838e7b1	lib/httpserver: properly set status code for empty response	2020-05-24 23:55:55 +03:00
Aliaksandr Valialkin	64bec11c91	lib/httpserver: fix compression for static files	2020-05-24 22:16:51 +03:00
Aliaksandr Valialkin	b747362936	lib/promscrape: mention about -promscrape.maxScrapeSize in the error message when target returns too big response	2020-05-24 14:41:24 +03:00
Aliaksandr Valialkin	be7253c084	lib/httpserver: do not recompress already compressed response ... This shoud help with vmauth issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/514	2020-05-22 16:45:20 +03:00
Aliaksandr Valialkin	b59e089ac7	app/vmagent: add -dryRun option for checking all the configs mentioned in command-line flags without running vmagent ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362	2020-05-21 15:23:18 +03:00
Aliaksandr Valialkin	482bae8466	lib/promscrape: add -promscrape.config.dryRun flag for checking -promscrape.config for errors or unsupported options ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/508 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362	2020-05-21 14:54:32 +03:00
Aliaksandr Valialkin	73ec5cf460	lib/promscrape: add -promscrape.discovery.concurrency and -promscrape.discovery.concurrentWaitTime flags for tuning the number of concurrent requests to autodiscovery API servers at Consul or Kubernetes	2020-05-19 17:35:59 +03:00
Aliaksandr Valialkin	2a8f1e6931	lib/storage: do not increment vm_slow_metric_name_loads_total counter for metric_ids which shouldnt be prefetched, since this may mislead users	2020-05-16 10:23:39 +03:00
Aliaksandr Valialkin	dc16cdd1ca	lib/persistentqueue: a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/484	2020-05-16 09:32:30 +03:00
肖贝贝	c154a92d29	fix: fix vmagent multi queue may become one because sync bug (#484) ... Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-05-16 09:32:29 +03:00
Aliaksandr Valialkin	0f3d46810b	lib/backup: remove misleading -dst mention in error message ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/482	2020-05-15 17:13:27 +03:00
Aliaksandr Valialkin	e72518e8c6	lib/backup: donload only the remaining parts for partially downloaded files after vmrestore restart ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487	2020-05-15 17:03:25 +03:00
Aliaksandr Valialkin	1e5c1d7eaa	app/vmstorage: add vm_slow_metric_name_loads_total metric, which could be used as an indicator when more RAM is needed for improving query performance	2020-05-15 14:12:24 +03:00
Aliaksandr Valialkin	d6b9a49481	app/vmstorage: add vm_slow_row_inserts_total and vm_slow_per_day_index_inserts_total metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series	2020-05-15 13:46:57 +03:00
Aliaksandr Valialkin	a72f18e821	lib/{storage,mergeset}: further tuning of compression levels depending on block size ... This should improve performance for querying newly added data, since it can be unpacked faster.	2020-05-15 13:12:28 +03:00
Aliaksandr Valialkin	2cf2e9955b	lib/storage: wait for all the goroutines to finish in TestSearch in order to prevent racy behavior on test finish	2020-05-15 12:12:20 +03:00
Aliaksandr Valialkin	67e331ac62	lib/storage: optimize ingestion pefrormance for new time series	2020-05-15 12:12:19 +03:00
Aliaksandr Valialkin	6838fa876c	lib/mergeset: tune compression levels in order to improve ingestion performance a bit	2020-05-15 12:12:15 +03:00
Aliaksandr Valialkin	1b5d272e07	lib/storage: reduce indentation in Storage.add	2020-05-14 23:23:56 +03:00
Aliaksandr Valialkin	71d29a8fa1	lib/storage: return the first error instead of the last error, since the first error usually points to the root cause	2020-05-14 23:18:59 +03:00
Aliaksandr Valialkin	3845420a8f	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin	7e831741f9	lib/{storage,mergeset}: return dst on error from unmarshalBlockHeaders, so it could be reused	2020-05-14 15:32:23 +03:00
Aliaksandr Valialkin	2f42b85e0e	lib/storage: document that getnerateUniqueMetricID should return dense ids	2020-05-14 14:08:59 +03:00
Aliaksandr Valialkin	f442d81648	lib/{storage,mergeset}: cleanup: remove unused partSearch.indexBlockReuse	2020-05-14 14:03:15 +03:00
Aliaksandr Valialkin	8bb44a5d09	lib/storage: optimize label matching for regexp ending with literal suffix ... For example, `{label=~"foo.*bar.+baz"}` contains literal suffix `baz`, so it should work faster now.	2020-05-13 11:39:05 +03:00
Aliaksandr Valialkin	3b0f66a227	app/vmagent: fix a bug with improper relabeling when multiple -remoteWrite.urlRelableConfig args are set ... This bug could result in incorrect relabeling and metrics' drop. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467	2020-05-12 22:03:45 +03:00
Aliaksandr Valialkin	c9ab6dc532	lib/fs: do not use mmap for 32-bit arches by default, since they cannot map files bigger than 4GB in RAM	2020-05-12 20:21:39 +03:00
Aliaksandr Valialkin	d54a93fc81	app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2020-05-12 17:23:43 +03:00
Aliaksandr Valialkin	405cf44aed	app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set	2020-05-12 15:24:53 +03:00
Aliaksandr Valialkin	bd5f4e0344	lib/storage: properly initialize part struct before trying to close it on error ... This should prevent from nil pointer dereference bug at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/468 .	2020-05-12 14:54:16 +03:00
Aliaksandr Valialkin	f7753b1469	lib/storage: gradually pre-populate per-day inverted index for the next day ... This should prevent from CPU usage spikes at 00:00 UTC every day when inverted index for new day must be quickly created for all the active time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430	2020-05-12 12:13:32 +03:00
Aliaksandr Valialkin	8c77cb436a	lib/storage: typo fixes in error messages: or -> of	2020-05-12 12:12:33 +03:00
Aliaksandr Valialkin	bbf06a4248	lib/storage: speed up matching for common regexps in label filters ... The following regexps have been optimized: * 'foo.+bar' * 'foo.+bar.+baz' This should improve performance for matching Graphite-like metrics.	2020-05-11 22:49:01 +03:00
Aliaksandr Valialkin	37254a139a	lib/storage: add a benchmark for Graphite-like regexps for metric names	2020-05-11 22:49:00 +03:00
Aliaksandr Valialkin	2f28e945b8	lib/httpserver: add -http.shutdownDelay flag for a grace period before http server shutdown ... The http server returns 503 non-OK error at `/health` page during grace period, so load balancers in front of the http server could re-route incoming requests to other servers. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/463	2020-05-07 15:25:51 +03:00
Aliaksandr Valialkin	3052b479b7	lib/httpserver: reduce typical duration for http server graceful shutdown ... Previously the duration for graceful shutdown for http server could take more than a minute because of imporperly set timeouts in setNetworkTimeout. Now typical duration for graceful shutdown should be reduced to less than 5 seconds.	2020-05-07 14:16:38 +03:00
Aliaksandr Valialkin	c43a265716	lib/flagutil: make errcheck happy by explicitly ignoring Array.Set result in tests	2020-05-06 22:37:28 +03:00
Aliaksandr Valialkin	15e3682b40	lib/flagutil: properly parse quoted flag values for flagutil.Array	2020-05-06 22:28:15 +03:00
Aliaksandr Valialkin	20538a2a5d	app/vmagent: allow setting independent auth configs per each configured -remoteWrite.url	2020-05-06 16:52:32 +03:00
Aliaksandr Valialkin	9f39e618ed	lib/promscrape/discovery/gce: discover per-zone instances for gce_sd_config in parallel. This should reduce discovery latency	2020-05-06 15:00:23 +03:00
Aliaksandr Valialkin	8ab5e47b5c	lib/promscrape: add Prometheus-compatible DNS-based service discovery aka dns_sd_configs	2020-05-06 00:02:41 +03:00
Aliaksandr Valialkin	42d563934b	lib/promscrape: properly connect to TCP6 addresses if -enableTCP6 is set	2020-05-06 00:02:40 +03:00
Aliaksandr Valialkin	1c8e97c8a0	lib/procutil: add NewSighupChan function, which returns a channel, which is triggered on every SIGHUP	2020-05-05 10:56:15 +03:00
Aliaksandr Valialkin	054457d1f4	lib/promscrape: allow explicitly setting empty token via token: "" in consul_sd_config	2020-05-05 07:49:54 +03:00
Aliaksandr Valialkin	89aa6dbf56	lib/promscrape: add Prometheus-compatible service discovery for Consul aka consul_sd_configs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330	2020-05-04 20:53:06 +03:00
Aliaksandr Valialkin	28e0e8fd88	lib/promauth: properly set up client certificate in tls.Config ... Previously the client certificate has been mistakenly set up as a server certificate	2020-05-04 20:53:04 +03:00
Aliaksandr Valialkin	ed91fe1d9b	lib/promscrape: move common code for discovery api config map handling into discoveryutils	2020-05-04 20:52:58 +03:00
Aliaksandr Valialkin	c50fd219dc	lib/promscrape/discovery/kubernetes/: unify apiConfig creation	2020-05-04 20:52:53 +03:00
Aliaksandr Valialkin	a5880f17af	lib/promscrape: remove debug line left after the commit e4aac6ea40	2020-05-03 17:16:19 +03:00
Aliaksandr Valialkin	1f0e8fdc0d	lib/promscrape: fix tests after the commit 658a8742ac ... The original commit copies `__address__` label to `instance` label when generating per-target labels as Prometheus does. See https://www.robustperception.io/life-of-a-label for details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/453	2020-05-03 16:59:29 +03:00
DexterZhang	317688f144	fix(vmagent): different behavior as how prometheus deal with labels. [Issue#453] (#454)	2020-05-03 16:59:28 +03:00
Aliaksandr Valialkin	ab1e6a76bb	lib/promscrape: make consistent scrape time offsets across reloads for the same ScrapeURL and Labels ... This should make consistent intervals between data points for scrape targets across reloads. Previously these intervals were random.	2020-05-03 14:31:22 +03:00
Aliaksandr Valialkin	f25416984b	lib/promscrape: fix TestGetFileSDScrapeWorkSuccess after 3b234d82e5	2020-05-03 14:31:20 +03:00
Aliaksandr Valialkin	f422203e10	lib/promscrape: reload only modified scrapers on config changes ... This should improve scrape stability when big number of targets are scraped and these targets are frequently changed. Thanks to @xbsura for the idea and initial implementation attempts at the following pull requests: - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/449 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/458 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/459 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/460	2020-05-03 12:47:16 +03:00
Aliaksandr Valialkin	bbaca16ce8	lib/httpserver: rename http.externalURL to http.pathPrefix and improve help message for this flag ... The `http.externalURL` flag name was slightly misleading, so it has been renamed to `http.pathPrefix`.	2020-05-02 13:12:24 +03:00
DexterZhang	a0589f2ca5	feat(httpserver): add http.externalUrl config to http server, it adds prefix to http path automatically (#452)	2020-05-02 13:12:23 +03:00
Aliaksandr Valialkin	b21b73115a	app/vminsert: add /-/reload handler in the same way as for vmagent	2020-04-30 02:18:08 +03:00
Aliaksandr Valialkin	a970705d8e	lib/procutil: prevent from app termination on SIGHUP signal, since this signal is frequently used for config reload	2020-04-30 02:18:06 +03:00
Aliaksandr Valialkin	d99f48aa48	lib/httpserver: mention that -http.maxGracefulShutdownDuration command-line flag value can be increased on shutdown timeout	2020-04-30 01:37:02 +03:00
Aliaksandr Valialkin	de5f923476	lib/promscrape: set 30 seconds timeout for discovery api requests ... Previously such requests could hang for long time. This could make debugging harder.	2020-04-29 17:29:03 +03:00
Aliaksandr Valialkin	b6d88bac04	vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp ... The upstream fasthttp may contain issues like 996610f021 , plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.	2020-04-29 16:43:09 +03:00
Aliaksandr Valialkin	9ed4951ec8	lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics	2020-04-28 15:30:06 +03:00
Aliaksandr Valialkin	d78ed50edd	lib/storage: recover when metricID->metricName entry is missing in the inverted index after unclean shutdown ... Newly added index entries can be missing after unclean shutdown, since they didn't flush to persistent storage yet. Log about this and delete the corresponding metricID, so it could be re-created next time.	2020-04-28 12:01:32 +03:00
Aliaksandr Valialkin	53740d0026	lib/promscrape: handle connection reset when targets responds with http redirect	2020-04-28 02:14:32 +03:00
肖贝贝	3e6f29f462	fix: vmagent not follow 301/302 redirect bug (#445) ... Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-04-28 02:14:31 +03:00
Aliaksandr Valialkin	424068f804	lib/promscrape: handle connection reset when targets responds with http redirect	2020-04-28 02:14:26 +03:00
肖贝贝	7d045bf2ca	fix: vmagent not follow 301/302 redirect bug (#445) ... Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-04-28 02:14:25 +03:00
Aliaksandr Valialkin	2aecf7c37c	lib/{encoding,decimal}: typo fixes in tests: epxecting->expecting	2020-04-28 00:02:19 +03:00
Aliaksandr Valialkin	806dc73d8a	lib/encoding: reduce possibility of failure in TestMarshalInt64ArraySize	2020-04-28 00:02:18 +03:00
Aliaksandr Valialkin	a603a15757	lib/promscrape/discovery/gce: make golangci-lint happy	2020-04-27 19:29:42 +03:00
Aliaksandr Valialkin	86a1d9cb0c	lib/promscrape: add initial support for Prometheus-compatible service discovery for Amazon EC2 aka ec2_sd_configs	2020-04-27 19:29:22 +03:00
Aliaksandr Valialkin	1acb6eb25a	lib/promscrape/discovery/gce: properly set filter query arg in api url	2020-04-27 16:01:53 +03:00
Aliaksandr Valialkin	0daa37fa02	lib/promscrape/discovery/gce: allow empty project and zone for gce_sd_config	2020-04-27 11:45:45 +03:00
Aliaksandr Valialkin	989d84cf3f	app/{vminsert,vmstorage}: wait for ack from vmstorage after each packet sent to it from vminsert ... This should protect from possible data loss when `vmstorage` is stopped while the packet is sent from `vminsert`. This commit switches to new protocol between vminsert and vmstorage, which is incompatible with the previous protocol. So it is required that both vminsert and vmstorage nodes are updated.	2020-04-27 09:53:26 +03:00
Aliaksandr Valialkin	e933cbac16	lib/storage: postpone reading data from blocks during search ... This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics during heavy queries, which touch big number of time series over long time ranges. This improves single-node VM performance on heavy queries by up to 2x.	2020-04-27 08:44:01 +03:00
Aliaksandr Valialkin	31861c5b8e	lib/promscrape/discovery/gce: allow empty zone arg in gce_sd_config - in this case zones for the given project are automatically discovered	2020-04-26 14:37:38 +03:00
Aliaksandr Valialkin	b16e19c053	lib/storage/dedup.go: go fmt	2020-04-26 14:37:36 +03:00
Aliaksandr Valialkin	a0000c3a6e	lib/storage: improve deduplication algorithm ... Now it leaves only the first data point on each `-dedup.minScrapeInterval` interval. Previously it may leave two data points on the interval. This could lead to unexpected results for `histogram_quantile(phi, sum(rate(buckets)) by (le))` query.	2020-04-26 13:10:18 +03:00
Aliaksandr Valialkin	13b4069c59	lib/storage: postpone label filters matching too many time series instead of giving up with error ... This should reduce the frequency of the following errors: cannot find tag filter matching less than N time series; either increase -search.maxUniqueTimeseries or use more specific tag filters more than N time series found on the time range [...]; either increase -search.maxUniqueTimeseries or shrink the time range	2020-04-24 21:18:52 +03:00
Aliaksandr Valialkin	7c74efd640	lib/promscrape/discovery/gce: make golint happy by ignoring resp.Body.Close() result	2020-04-24 18:13:26 +03:00
Aliaksandr Valialkin	069690e3bd	lib/promscrape: initial implementation for gce_sd_configs aga Prometheus-compatible service discovery for Google Compute Engine	2020-04-24 17:53:43 +03:00
Aliaksandr Valialkin	de991551f5	lib/promscrape: query /api/v1/namespaces/* for the configured namespaces in kubernetes_sd_config ... This should fix authroization issues described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432	2020-04-24 14:42:02 +03:00
Aliaksandr Valialkin	387a21c96d	lib/promscrape: add -promscrape.configCheckInterval command-line flag for automating config checking ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/431	2020-04-23 23:41:26 +03:00
Aliaksandr Valialkin	83e4c8427e	lib/promscrape: access Config entries by reference, so they can be compared by addresses	2020-04-23 14:38:29 +03:00
Aliaksandr Valialkin	e220f3eeb6	lib/promscrape: move KubernetesSDConfig to lib/promscrape/discovery/kubernetes	2020-04-23 11:34:30 +03:00
Aliaksandr Valialkin	1187494c8f	lib/promscrape/discovery/kubernetes: hide role switch logic behind GetLabels function	2020-04-22 22:16:18 +03:00
Aliaksandr Valialkin	f9526809e5	app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues ... See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268	2020-04-22 22:03:23 +03:00
Aliaksandr Valialkin	f3e5722257	lib/writeconcurrencylimiter: improve docs for -maxConcurrentInserts command-line flag	2020-04-20 21:03:09 +03:00
Aliaksandr Valialkin	81481abaa9	lib/promscrape/discovery/kubernetes: reuse a client for empty api_server inside different jobs	2020-04-20 17:07:37 +03:00
Aliaksandr Valialkin	6764efde39	lib/promscrape/discovery/kubernetes: update stale comments	2020-04-17 14:06:26 +03:00
Aliaksandr Valialkin	d86640d609	lib/promscrape: suppress scrape errors if -promscrape.suppressScrapeErrors flag is set	2020-04-16 23:41:52 +03:00
Aliaksandr Valialkin	70104f3fb1	lib/promscrape: print all the labels for the target on error message for failed scrape ... This should improve debuggability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/420	2020-04-16 23:35:10 +03:00
Aliaksandr Valialkin	266bbec52d	lib/promscrape: retry target scraping when the target closes previously established keep-alive connection to it ... This should fix the following error: the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection	2020-04-16 23:25:34 +03:00
Aliaksandr Valialkin	b2d009c8db	lib/logger: typo fix	2020-04-16 00:20:02 +03:00
Aliaksandr Valialkin	d4bc60d63c	lib/logger: add WARN level for logging expected errors such as invalid user queries	2020-04-15 20:50:45 +03:00
Aliaksandr Valialkin	a873b553cf	app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415	2020-04-15 12:01:05 +03:00
Aliaksandr Valialkin	99f0cb1f5f	lib/promscrape: code cleanup in runScraper func	2020-04-15 11:36:35 +03:00
Aliaksandr Valialkin	e9d9638627	lib/storage: skip metricID if the corresponding metricID->metricName is missing in inverted index during search ... This case is possible when the corresponding metricID->metricName entry didn't propagate to inverted index yet. This should fix the following error: error when searching tsids for tfss [...]: cannot find metricName by metricID 1582417212213420669: EOF	2020-04-15 00:10:11 +03:00
Aliaksandr Valialkin	6ec582acb9	lib/promscrape: show information on improperly configured scrape targets at the bottom of /targets page ... This is a common error whith improperly configured target autodiscovery and/or relabeling. This error leads to duplicate scraping of the same targets with the same set of labels, which leads to duplicate samples in time series.	2020-04-14 14:55:13 +03:00
Aliaksandr Valialkin	391fb0903e	lib/promscrape/discovery/kubernetes: remove only unused client for API server during cleaning	2020-04-14 14:19:26 +03:00
Aliaksandr Valialkin	636e1578de	lib/promscrape: add promrelabel.GetLabelValueByName helper function	2020-04-14 14:12:15 +03:00
Aliaksandr Valialkin	3945bf9dec	lib/promscrape: mention job name in error messages when target cannot be scraped ... This should improve debuggability	2020-04-14 13:33:18 +03:00
Aliaksandr Valialkin	66da177fe9	lib/promscrape: reset ScrapeWork.ID in tests	2020-04-14 13:31:37 +03:00
Aliaksandr Valialkin	88366cad15	lib/promscrape: properly expose statuses for targets with duplicate scrape urls at /targets page ... Previously targets with duplicate scrape urls were merged into a single line on the page. Now each target with duplicate scrape url is displayed on a separate line.	2020-04-14 13:10:06 +03:00
Aliaksandr Valialkin	09f796e2ab	lib/promscrape: remove labels starting with __meta_ after applying relabel_configs as Prometheus does ... This should reduce CPU load during scraping when target discovery generates big number of `__meta_*` labels (for instance, k8s discovery). See https://www.robustperception.io/life-of-a-label for details.	2020-04-14 12:23:30 +03:00
Aliaksandr Valialkin	f58d15f27c	lib/promscrape: rename 'scrape_config->scrape_limit' to 'scrape_config->sample_limit' ... `scrape_config` block from Prometheus config contains `sample_limit` field, while in `vmagent` this field was mistakenly named as `scrape_limit`.	2020-04-14 12:00:03 +03:00
Aliaksandr Valialkin	7c4fb038e3	lib/promscrape: add initial support for kubernetes_sd_config ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334	2020-04-13 21:03:53 +03:00
Aliaksandr Valialkin	4017163393	lib/promscrape: add -promscrape.config.strictParse flag for detecting errors in -promscrape.config file	2020-04-13 13:15:52 +03:00
Aliaksandr Valialkin	7fbfef2aee	lib/promscrape: extract common auth code to lib/promauth	2020-04-13 12:59:22 +03:00
Aliaksandr Valialkin	e0c6da8e2a	lib/storage: disable deduplication after dedup tests are complete ... The rest of tests expect that the de-duplication is disabled.	2020-04-10 17:33:38 +03:00
Aliaksandr Valialkin	8ed0d5471a	lib/storage: correctly handle -dedup.minScrapeInterval values smaller than 8ms ... Such small values may be used for removing samples with duplicate timestamps. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/409 for details.	2020-04-10 16:40:41 +03:00
Aliaksandr Valialkin	0b2f678d8e	lib/{storage,mergeset}: make sure that requests and misses cache counters never go down	2020-04-10 14:44:52 +03:00
Aliaksandr Valialkin	661cfb03e2	lib/protoparser: add -*TrimTimstamp command-line flags for Influx, Graphite, OpenTSDB and CSV data ... These flags can be used for reducing disk space usage for timestamps data ingested over the given protocols	2020-04-10 12:44:46 +03:00
Aliaksandr Valialkin	f0b08dbd9e	lib/workingsetcache: accumulate stat counters on cache rotation ... This should prevent from cache stats counters going down after cache rotation, which may corrupt `cache hit ratio` graph on the official Grafan dasbhoards when using the following query: 1 - (sum(rate(vm_cache_misses_total[5m])) by (type) / sum(rate(vm_cache_requests_total[5m])) by (type))	2020-04-10 11:51:47 +03:00
Aliaksandr Valialkin	28c65b58a2	lib/memory: add more details to -memory.allowedPercent help message	2020-04-09 15:34:21 +03:00
Aliaksandr Valialkin	84fa146792	lib/httpserver: remove unnecessary http.HandlerFunc wrapper in gzipHandler	2020-04-01 18:14:47 +03:00
Aliaksandr Valialkin	0ad7aaf535	lib/storage: add missing reset for tagFilter.matchesEmptyValue on tagFilter.Init	2020-04-01 17:40:27 +03:00
Aliaksandr Valialkin	c189104be7	lib/promscrape: reduce timestamp jitter when scraping targets ... This should improve compression for timestamps	2020-04-01 16:13:01 +03:00
Aliaksandr Valialkin	4c56acbafa	lib/storage: remove duplicate data points on 7/8*minScrapeInterval interval instead of 1/2*minScrapeInterval ... This should reduce storage usage and should improve deduplication accuracy	2020-04-01 15:47:04 +03:00
Aliaksandr Valialkin	504ea876f2	lib/storage: handle errors returned from TagFilters.Add when cloning TagFilters with negative filter	2020-03-31 16:18:34 +03:00
Aliaksandr Valialkin	ef714e01c1	lib/storage: add fast path for the previous indexdb search if it doesn't contain per-day inverted index yet	2020-03-31 12:35:15 +03:00
Aliaksandr Valialkin	7e755b4bac	lib/storage: optimize per-day inverted index search for tag filters matching big number of time series ... - Sort tag filters in the ascending number of matching time series in order to apply the most specific filters first. - Fall back to metricName search for filters matching big number of time series (usually this are negative filters or regexp filters).	2020-03-31 00:53:29 +03:00
Aliaksandr Valialkin	d450249955	lib/storage: properly handle {label=~"foo|"} filters as Prometheus does ... Such filters must match all the time series with `label="foo"` plus all the time series without `label` Previously only time series with `label="foo"` were matched.	2020-03-30 20:21:47 +03:00
Aliaksandr Valialkin	b47444e69d	lib/envflag: add -envflag.prefix for setting optional prefix for environment vars	2020-03-30 15:51:44 +03:00
kreedom	f058efb3d1	[vmalert] config parser (#393) ... * [vmalert] config parser * make linter be happy * fix test * fix sprintf add test for rule validation	2020-03-29 01:49:40 +02:00
Aliaksandr Valialkin	42c290ce9f	lib/httpserver: add -http.maxGracefulShutdownDuration command-line flag for tuning the maximum duration required for graceful shutdown of http server	2020-03-27 20:10:05 +02:00
Aliaksandr Valialkin	8fa80a2dbc	lib/uint64set: remove zero buckets after Set.Intersect	2020-03-27 01:16:34 +02:00
Aliaksandr Valialkin	7a35447031	lib/uint64set: small code cleanup and perf tuning ... * Remember the last accessed bucket on Has() call. * Inline fast paths inside Add() and Has() calls. * Remove fragile code with maxUnsortedBuckets inside bucket32.	2020-03-25 15:29:59 +02:00
Aliaksandr Valialkin	cce936de5b	lib/uint64set: go fmt	2020-03-24 22:28:09 +02:00
Aliaksandr Valialkin	7cdac6634c	lib/storage: serialize snapshot creation process with mutex ... This guarantees that the snapshot contains all the recently added data from inmemory buffers when multiple concurrent calls to Storage.CreateSnapshot are performed.	2020-03-24 22:27:28 +02:00
Aliaksandr Valialkin	c31b956355	lib/uint64set: added more tests	2020-03-24 22:27:26 +02:00
Aliaksandr Valialkin	31a533656e	lib/storage: remove obsolete code	2020-03-13 22:42:42 +02:00
Aliaksandr Valialkin	bf1869d33d	lib/promscrape: allow overriding external_labels as Prometheus does ... Prometheus docs at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config say: > In communication with external systems, they are always applied only > when a time series does not have a given label yet and are ignored otherwise. Though this may result in consistency chaos when scrape targets override `external_labels`, let's stick with Prometheus behavior for the sake of backwards compatibility. There is last resort in vmagent with `-remoteWrite.label`, which consistently sets the configured labels to all the metrics before sending them to remote storage. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/366	2020-03-12 20:25:27 +02:00
Aliaksandr Valialkin	0e7a71a245	app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369	2020-03-12 19:13:56 +02:00
Aliaksandr Valialkin	fa7910fba1	lib/protoparser/csvimport: add missing metric vm_rows_invalid_total{type="csvimport"}	2020-03-12 15:28:10 +02:00
Aliaksandr Valialkin	375d5483fa	lib/promscrape: remove possible races when registering and de-registering scrape workers for /targets page	2020-03-11 16:30:43 +02:00
Aliaksandr Valialkin	187fd89c70	lib/promscrape: consistently update /targets page after SIGHUP	2020-03-11 03:20:38 +02:00
Aliaksandr Valialkin	2f0a36044c	app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv	2020-03-10 21:17:40 +02:00
Aliaksandr Valialkin	7545784a49	all: fix golangci-lint issues	2020-03-10 19:40:03 +02:00
Aliaksandr Valialkin	d39dd8aa69	lib/promscrape: do not retry idempotent requests when scraping targets ... This should prevent from the following unexpected side-effects of idempotent request retries: - increased actual timeout when scraping the target comparing to the configured scrape_timeout - increased load on the target Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/357	2020-03-09 13:31:20 +02:00
Aliaksandr Valialkin	12789a4621	app/vmagent: do not allow non-supported fields in -remoteWrite.relabelConfig and file_sd_configs ... This should reduce possible confusion like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/363	2020-03-06 20:19:34 +02:00
Aliaksandr Valialkin	0d893eff36	Makefile: add build and test rules with enabled race detector. These rules have -race suffix ... Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .	2020-03-05 12:05:16 +02:00
Aliaksandr Valialkin	31a76a7b3a	lib/promscrape: consistency renaming: stopCh -> globalStopCh	2020-03-03 20:08:22 +02:00
Aliaksandr Valialkin	f01d1bf4a8	app/vmagent: add -remoteWrite.maxDiskUsagePerURL for limiting the maximum disk usage for each -remoteWrite.url buffer ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352	2020-03-03 19:49:20 +02:00
Aliaksandr Valialkin	c3b239eb1a	lib/protoparser/prometheus: allow trailing comma in tags list ... The trailing comma is generated by cloudwatch exporter. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/350	2020-03-02 22:23:28 +02:00
Aliaksandr Valialkin	6a1aab88fd	lib/protoparser: metrics renaming: vm_protoparser_<type>_* -> vm_protoparser_*{type="<type>"} ... This should improve composability of these metrics in PromQL queries	2020-02-28 20:19:59 +02:00
Aliaksandr Valialkin	c2e602286c	lib/persistentqueue: reset chunk file when the persistent queue is empty	2020-02-28 20:06:59 +02:00
Aliaksandr Valialkin	cf9aee4ec3	all: properly split vm_deduplicated_samples_total among cluster components ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345	2020-02-27 23:47:51 +02:00
Aliaksandr Valialkin	5e7b4795bd	lib/envflag: typo fix in docs to -envflag.enable: envoronment->environment	2020-02-27 21:56:46 +02:00
Aliaksandr Valialkin	e6a481ab11	lib/promscrape: properly reload new configs on SIGHUP	2020-02-26 13:54:24 +02:00
Aliaksandr Valialkin	fa6815712f	lib/promscrape: go fmt	2020-02-26 13:24:40 +02:00
Aliaksandr Valialkin	f2a6948a14	lib/promscrape: do not add missing port to __address__ label in order to be consistent with Prometheus behavior ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/331	2020-02-25 20:50:18 +02:00
Aliaksandr Valialkin	c4194020ef	app/vmagent: do not allow sending unpacked requests with sizes exceeding -maxInsertRequestSize	2020-02-25 19:35:43 +02:00
Aliaksandr Valialkin	2471340e0d	app/vmagent: add ability to accept Influx line protocol data via TCP and UDP ... Just set `-influxListenAddr` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333	2020-02-25 19:18:01 +02:00
Aliaksandr Valialkin	7a045125cc	lib/fs: typo fix: read blocks bigger than 8KB via pread() call instead of using mmap	2020-02-25 18:04:06 +02:00
Aliaksandr Valialkin	13ee8271d0	lib/envflag: substitute dots with underscores in env var names if -envflag.enable is set ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-24 21:15:11 +02:00
Aliaksandr Valialkin	7ee7614e90	app/vmagent: initial implementation for vmagent	2020-02-23 17:31:54 +02:00
Aliaksandr Valialkin	110cce24d9	lib/storage: add vm_ prefix to deduplicated_samples_total metric	2020-02-21 19:33:36 +02:00
Aliaksandr Valialkin	9d279e26a7	lib/protoparser/prometheus: skip leading whitespace from tag names	2020-02-16 19:06:23 +02:00
Aliaksandr Valialkin	a2b81b71b9	lib/storage: typo fix	2020-02-16 15:53:48 +02:00
Aliaksandr Valialkin	ad4cb9f3ca	lib/storage: prevent from clobbering nin-nil lastError in Storage.add	2020-02-16 15:51:35 +02:00
Aliaksandr Valialkin	846d7fa7e9	app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions ... This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.	2020-02-13 17:01:50 +02:00
Aliaksandr Valialkin	e3b18ca1ab	lib/mergeset: skip createing temporary part objects when merging source inmemory parts ... This should reduce CPU usage when adding new entries to inverted index. This should alos prevent from creating stalled cleaner goroutines for the created temporary parts, since they were never closed. This should fix the following issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .	2020-02-13 14:09:13 +02:00
Aliaksandr Valialkin	347aaba79d	lib/{storage,mergeset}: use time.Ticker instead of time.Timer where appropriate ... It has been appeared that time.Timer was used in places where time.Ticker must be used instead. This could result in blocked goroutines as in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .	2020-02-13 13:21:48 +02:00
Aliaksandr Valialkin	e7d1037210	docs: migrate ExtendedPromQL->MetricsQL in order to be more consistent	2020-02-10 23:03:31 +02:00
Aliaksandr Valialkin	fcdd95a6ef	lib/envflag: check for incorrect flag values read from environment vars	2020-02-10 16:09:03 +02:00
Aliaksandr Valialkin	9c5db9400c	lib/envflag: add -envflag.enable command-line flag for enabling reading flags from environment vars ... By default flags are read only from command line. They can be read from environment vars if `-envflag.enable` is set. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-10 16:09:01 +02:00
Aliaksandr Valialkin	1010a57882	all: allow setting flags via environment vars ... Now flags can be set via environment vars with the same names as flags. Command-line flags override flags set via env vars. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-10 13:31:21 +02:00
Aliaksandr Valialkin	ea66212c93	lib/storage: move -dedup.minScrapeInterval flag outside lib/storage, so it doesnt show up in vminsert in cluster version	2020-02-10 13:07:25 +02:00
Aliaksandr Valialkin	8b360a25e9	lib/logger: initialize output to os.Stderr by default	2020-02-04 22:43:26 +02:00
Aliaksandr Valialkin	1f271a9815	lib/logger: add -loggerOutput command-line flag ... This flag allows changing log output from `stderr` to `stdout` if `-loggerOutput=stdout` is set. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/306	2020-02-04 21:48:24 +02:00
Aliaksandr Valialkin	49ab3fa076	lib/logger: do not clutter -loggerFormat=json output with stack trace ... This should improve json parsing	2020-02-04 21:40:20 +02:00
Aliaksandr Valialkin	56d6b8ed0a	lib/storage: do not deduplicate blocks with less than 32 samples during merge ... This should improve deduplication accuracy for blocks with higher number of samples.	2020-02-04 18:41:37 +02:00
Aliaksandr Valialkin	7cde594696	all: do not clash flag description with back-quoted flag types ... See https://golang.org/pkg/flag/#PrintDefaults for more details.	2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin	9b25a2fb67	lib/fs: remove unused readerAt interface	2020-01-31 15:13:00 +02:00
Aliaksandr Valialkin	e3adc095bd	all: add -dedup.minScrapeInterval command-line flag for data de-duplication ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278	2020-01-31 01:18:54 +02:00
Aliaksandr Valialkin	a45f25699c	lib/storage: re-use indexSearch inside Storage.prefetchMetricNames	2020-01-31 01:18:53 +02:00
Aliaksandr Valialkin	cb5c39ee70	lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor ... This should improve performance when reading many small blocks.	2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin	da19fffa08	all: rename ReadAt* to MustReadAt* in order to dont clash with io.ReaderAt	2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin	1332ddc15e	lib/storage: pass missing AccountID and ProjectID to searchMetricName	2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin	4ed5e9a7ce	lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init ... This should speed up Search.NextMetricBlock loop for big number of found time series.	2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin	cb2a2f281f	lib/mergeset: properly update lastAccesstime in indexBlockCache entries ... This is a follow-up for 6665f10e7b	2020-01-29 21:21:01 +02:00
Aliaksandr Valialkin	170c1c3a4e	app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value	2020-01-29 00:48:14 +02:00
Aliaksandr Valialkin	a9c1d5b351	app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps	2020-01-28 22:53:50 +02:00
Aliaksandr Valialkin	81ba371eaf	lib/logger: fix improperly set skipframes for all the logging functions	2020-01-26 18:34:58 +02:00
Aliaksandr Valialkin	9f595cb2b1	lib/httpserver: log the caller of httpserver.Errorf ... Previously log message contained `httpserver.Errorf`, not it contains the caller of `httpserver.Errorf`, which is more useful.	2020-01-25 20:18:06 +02:00
Aliaksandr Valialkin	36a1a21d6e	lib/protoparser: add parser for Prometheus exposition text format ... This parser will be used by vmagent	2020-01-24 20:11:19 +02:00
Aliaksandr Valialkin	0cda6afa8e	app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent	2020-01-24 16:55:18 +02:00
Aliaksandr Valialkin	ea53a21b02	all: consistently log durations in seconds with millisecond precision ... This should improve logs readability	2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin	40e564eb9c	app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d	2020-01-21 19:05:29 +02:00
Aliaksandr Valialkin	9eaa2ab871	app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp	2020-01-21 15:00:35 +02:00
Aliaksandr Valialkin	62b041e90a	lib/{mergeset,storage}: properly update lastAccessTime in index and data block cache entries	2020-01-20 15:00:10 +02:00
Aliaksandr Valialkin	607d4418b8	lib/uint64set: add missing bucket32.b16his values	2020-01-18 14:26:23 +02:00
Aliaksandr Valialkin	e3379537cd	lib/uint64set: optimize Set.Union ... This should improve performance for queries over big number of time series	2020-01-18 13:47:34 +02:00
Aliaksandr Valialkin	5077efd3f7	lib/uint64set: add benchmarks for Set.Union	2020-01-18 13:47:33 +02:00
Aliaksandr Valialkin	a851c75703	lib/storage: skip recovering timestamps order for lossless compression (PrecisionBits=64)	2020-01-17 23:59:19 +02:00
Aliaksandr Valialkin	2084921e64	all: use github.com/klauspost/compress/gzip instead of compress/gzip ... `github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`. This gives better gzip compression and decompression speeds.	2020-01-17 23:59:17 +02:00
Aliaksandr Valialkin	ab4d5d72eb	lib/uint64set: reduce memory allocations in Set.AppendTo	2020-01-17 22:33:00 +02:00
Aliaksandr Valialkin	476c7fb109	lib/storage: reduce memory allocations when merging metricID sets	2020-01-17 22:10:56 +02:00
Aliaksandr Valialkin	29d21259f0	lib/uint64set: typo fix in Set.Intersect	2020-01-17 18:11:46 +02:00
Aliaksandr Valialkin	ed1d259b10	lib/uint64set: optimize Intersect, Subtract and Union functions ... This should improve performance for queries over big number of time series.	2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin	68d35357b1	lib/uint64set: improve benchmark for Set.Intersect	2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin	ffe352ad31	lib/uint64set: add benchmark for Set.Intersect	2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin	4b16b7fd11	all: mention command-line flags used for limiting the incoming request size in error messages ... This should improve error logs usability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287	2020-01-16 13:06:43 +02:00
Aliaksandr Valialkin	7d429e2806	lib/uint64set: reduce memory usage in Union, Intersect and Subtract methods ... Iterate items with newly added Set.ForEach method instead of allocating `[]uint64` slice for all the items before the iteration.	2020-01-15 12:15:48 +02:00
Aliaksandr Valialkin	caffb0cd01	lib/{mergeset,storage}: fix uint64 counters alignment for 32-bit architectures (GOARCH=386, GOARCH=arm)	2020-01-14 22:47:42 +02:00
Aliaksandr Valialkin	b03ccbf6f7	lib/{storage,mergeset}: gradually remove stale entries from block cache and index caches ... This should reduce memory usage in the long run when old blocks and indexes aren't accessed anymore.	2020-01-14 21:38:29 +02:00
Aliaksandr Valialkin	bcd3f0c5bd	app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions ... These functions can be used for calculating Hoeffding bounds for `m` over `d` time range and for the given `phi` in the range `[0..1]`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283	2020-01-11 14:47:13 +02:00
Aliaksandr Valialkin	87a106702b	app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d]) function ... This function can be used for simultaneous calculating of multiple `aggr_func*` functions that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])` would calculate `min_over_time` and `max_over_time` for `m[d]`.	2020-01-10 21:18:12 +02:00
Aliaksandr Valialkin	c314d9a219	app/vmselect/promql: add tmin_over_time(m[d]) and tmax_over_time(m[d]) functions ... These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`	2020-01-10 19:39:34 +02:00
Aliaksandr Valialkin	1029b6ab34	lib/backup/s3remote: check whether the file exists before deleting it ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/284	2020-01-09 23:20:51 +02:00
Aliaksandr Valialkin	705af61587	app/{vmbackup,vmrestore}: add backup complete file to backup when it is complete and check for this file before restoring from backup ... This should prevent from restoring from incomplete backups. Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.	2020-01-09 15:35:45 +02:00
Aliaksandr Valialkin	53e176ed67	lib/storage: limit maxRaRowsPerPartition by 500K for any number of rawRowsShardsPerPartition ... This should reduce write amplification for high ingestion rate on multi-CPU systems	2020-01-04 23:58:23 +02:00
Aliaksandr Valialkin	89b551201c	lib/metricsql: export IsRollupFunc and IsTransformFunc, since they can be used by package users	2020-01-04 13:25:13 +02:00
Aliaksandr Valialkin	6f29d37cb5	app/vmselect/promql: add histogram_share(le, buckets) function	2020-01-04 12:53:08 +02:00
Aliaksandr Valialkin	2290503140	app/vmselect/promql: add absent_over_time(m[d]) func similar to the function in Prometheus 2.16 ... See https://github.com/prometheus/prometheus/issues/2882	2020-01-04 12:53:01 +02:00
Aliaksandr Valialkin	67f94bbe12	app/vmselect/promql: add histogram_over_time(m[d]) rollup function	2020-01-04 12:52:56 +02:00
Aliaksandr Valialkin	588531dd76	lib/uint64set: reduce memory usage when storing big number of sparse metric_id values	2020-01-03 18:17:17 +02:00
Aliaksandr Valialkin	e0abf45d45	app/vmselect/promql: add share_le_over_time and share_gt_over_time functions for SLI and SLO calculations	2020-01-03 00:41:36 +02:00
Aliaksandr Valialkin	19962e2732	docs: refer to standalone MetricsQL package	2020-01-02 23:43:43 +02:00
Aliaksandr Valialkin	0d2e83e9d7	lib/metricsql: add example for ExpandWithExprs	2019-12-26 21:31:15 +02:00
Aliaksandr Valialkin	eb1a66c577	lib/metricsq: add ExpandWithExprs	2019-12-25 22:20:21 +02:00
Aliaksandr Valialkin	453d71d082	Rename lib/promql to lib/metricsql and apply small fixes	2019-12-25 22:09:09 +02:00
Mike Poindexter	009d1559db	Split Extended PromQL parsing to a separate library	2019-12-25 22:09:07 +02:00
Aliaksandr Valialkin	f22c9dbb0f	lib/fs: typo fix in fadvise_unix.go	2019-12-24 21:00:04 +02:00
Aliaksandr Valialkin	d3c185f0ca	lib/encoding: log the compressed block contents if it cannot be decompressed or unmarshaled ... This should help detecting the root cause of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281	2019-12-24 20:48:25 +02:00
Aliaksandr Valialkin	091e35cf0c	lib/encoding: mention src contents in error message returned from unmarshalInt64NearestDelta* ... This should simplify detecting the root cause of the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281	2019-12-24 20:41:38 +02:00
Aliaksandr Valialkin	0e51058a0d	lib/encoding: mention unpacked block size in the error message if unparsed tail left ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281	2019-12-24 20:35:20 +02:00
Aliaksandr Valialkin	afa8b34d27	Revert "lib/logger: prevent from blocking when log output isn't consumed in timely manner" ... This reverts commit 9f50232e70. Reason to revert: this leaves incomplete logs on app shutdown.	2019-12-24 12:20:45 +02:00
Aliaksandr Valialkin	6358cf3d47	app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs	2019-12-23 23:16:26 +02:00
Aliaksandr Valialkin	44f886cc9c	lib/encoding/zstd: typo fix	2019-12-23 18:37:20 +02:00
Aliaksandr Valialkin	108a60d69e	lib/encoding/zstd: call zstd.Decoder.Close instead of zstd.Decoder.Reset in order to free up occupied goroutines ... This should fix goroutine leak for https://github.com/klauspost/compress/issues/195	2019-12-23 18:32:28 +02:00
Aliaksandr Valialkin	335bd0ac0a	lib/encoding/zstd: prevent from possible encoder leak when concurrent goroutines create encoders for the same compressionLevel ... Thanks to @klauspost for the pointer to this issue. See https://github.com/klauspost/compress/issues/195 for details.	2019-12-23 18:06:02 +02:00
Aliaksandr Valialkin	9f50232e70	lib/logger: prevent from blocking when log output isn't consumed in timely manner ... Drop log messages instead of blocking and increment `vm_log_messages_dropped_total` metric.	2019-12-20 11:49:42 +02:00
Aliaksandr Valialkin	a37a006f11	lib/storage: scale ingestion performance by sharding rawRows on systems with more than 8 CPU cores	2019-12-19 18:17:05 +02:00
Aliaksandr Valialkin	8d79412b26	lib/storage: optimize bulk import performance when multiple data points are inserted for the same time series ... This should speed up `/api/v1/import` and make it more scalable on multi-core systems.	2019-12-19 15:13:36 +02:00
Aliaksandr Valialkin	05ec8afb3a	lib/httpserver: sync the code with master branch	2019-12-18 23:08:32 +02:00
Aliaksandr Valialkin	a7bf8e77af	app/vminsert: simultaneously accept telnet put and HTTP /api/put OpenTSDB metrics at -opentsdbListenAddr ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266	2019-12-14 00:42:18 +02:00
Aliaksandr Valialkin	bc3984a5b3	lib/logger: add -loggerFormat for choosing log message formats ... Supported formats: default, json Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/265	2019-12-13 15:09:18 +02:00
Aliaksandr Valialkin	5d2ff573aa	app/vmselect/promql: allow negative offsets ... Updates https://github.com/prometheus/prometheus/issues/6282	2019-12-11 00:57:51 +02:00
Aliaksandr Valialkin	3694efd005	lib/{mergeset,storage}: log info message when both source and destination part paths from txn are missing during startup ... This is expected condition after unclean shutdown (OOM, hard reset, `kill -9`) on NFS disk.	2019-12-09 15:45:23 +02:00
Aliaksandr Valialkin	639967db59	lib/{mergeset,storage}: make sure pending transaction deletions are finished before and after runTransactions call. ... `runTransactions` call issues async deletions for transaction files. The previously issued transaction deletions can race with the next call to `runTransactions`. Prevent this by waiting until all the pending transaction deletions are funished in the beginning of `runTransactions`. Also make sure that all the pending transaction deletions are finished before returning from `runTransactions`.	2019-12-04 21:40:52 +02:00
Aliaksandr Valialkin	7c0dd85a7c	lib/httpserver: add /ping handler for compatibility with Influx agents ... Certain Influx agents check for `/ping` endpoint before starting to send Influx line protocol data. See https://docs.influxdata.com/influxdb/v1.7/tools/api/#ping-http-endpoint	2019-12-04 19:18:18 +02:00
Aliaksandr Valialkin	534da0a8c3	lib/storage: fall back to global inverted index if a filter match too many time series in per-day index ... Previously this resulted to error message. The query may succeed via search in global index.	2019-12-03 14:48:08 +02:00
Aliaksandr Valialkin	6eb698d1cc	lib/storage: fix printing tag filters in TagFilters.String	2019-12-03 14:25:20 +02:00
Aliaksandr Valialkin	c04f60db35	lib/storage: print __name__ instead of empty string in user-visible tag filters	2019-12-03 14:18:18 +02:00
Aliaksandr Valialkin	625f6ca761	lib/storage: optimize regexp filter search	2019-12-03 00:33:53 +02:00
Aliaksandr Valialkin	b9616c017f	lib/{mergeset,storage}: remove transaction files only after the mentioned dirs are really removed ... This should fix the issue on NFS when incompletely removed dirs may be left after unclean shutdown (OOM, kill -9, hard reset, etc.), while the corresponding transaction files are already removed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-12-02 21:34:37 +02:00
Aliaksandr Valialkin	4e22b521c2	lib/storage: remove metricID with missing metricID->metricName entry ... The metricID->metricName entry can be missing in the indexdb after unclean shutdown when only a part of entries for new time series is written into indexdb. Recover from such a situation by removing the broken metricID. New metricID will be automatically created for time series with the given metricName when new data point will arive to it.	2019-12-02 20:52:13 +02:00
Aliaksandr Valialkin	5a62415bec	lib/storage: protect from time drift during indexdb rotation ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/248	2019-12-02 14:43:11 +02:00
Aliaksandr Valialkin	cf85c567d1	lib/logger: merge file and line labels into location="file:line" ... This should improve the usability for `vm_log_messages_total` metric during practical queries	2019-12-02 14:43:09 +02:00
Aliaksandr Valialkin	f055dbefda	lib/storage: generate more human-friendly result in TagFilters.String	2019-12-02 13:56:40 +02:00
Aliaksandr Valialkin	29f39f866e	lib/logger: consistency renaming from vm_log_messages_count to vm_log_messages_total, since this is a counter	2019-12-02 00:47:12 +02:00
Aliaksandr Valialkin	15eaff1745	lib/logger: track the number of log messages by (level, file, line) in the vm_log_messages_count metric	2019-12-01 18:38:30 +02:00
Aliaksandr Valialkin	d456ec7589	lib/netutil: use IPv6 for both listening and dialing if -enabledTCP6 is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/244	2019-12-01 02:52:53 +02:00
Aliaksandr Valialkin	7e734433a3	lib/backup: cosmetic fixes after #243	2019-11-29 18:07:41 +02:00
glebsam	4a192cb832	Add option to provide custom endpoint for S3, add option to specify S3 config profile (#243) ... * Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile * make fmt	2019-11-29 18:07:39 +02:00
Aliaksandr Valialkin	4810f1dde6	lib/netutil: add -enableTCP6 command-line flag for enabling listening for IPv6 additionally to IPv4 TCP ports	2019-11-29 17:33:07 +02:00
Aliaksandr Valialkin	409c939621	lib/backup: remove flock.lock file in empty dirs ... This fixes an issue when VictoriaMetrics doesn't see the restored data after the following operations: 1. Stop VictoriaMetrics. 2. Delete `<-storageDataPath>` dir. 3. Start VictoriaMetrics, then stop it. 4. Restore data from backup with `vmrestore`. 5. Start VictoriaMetrics. `vmrestore` didn't delete properly empty dirs in `<-storageDataPath>/indexdb` because of the remaining `flock.lock` files in these dirs.	2019-11-28 13:39:28 +02:00
Aliaksandr Valialkin	0f184affa7	app/vmselect/promql: optimize binary search over big number of samples during rollup calculations	2019-11-25 14:01:54 +02:00
Aliaksandr Valialkin	d24fc87a6f	lib/decimal: calculate ln2/ln10 constant during compile time	2019-11-23 15:52:39 +02:00
Aliaksandr Valialkin	2af7ca1122	vendor: update github.com/VictoriaMetrics/metrics from v1.7.2 to v1.8.0. This version supports histograms	2019-11-23 00:21:57 +02:00
Aliaksandr Valialkin	b9e53490b9	lib/storage: move non-matching tag filters to the top at matchTagFilters ... This should reduce the amount of useless work needed for matching the next metricNames.	2019-11-21 21:40:36 +02:00
Aliaksandr Valialkin	33d9d63393	lib/storage: speed up time series search for queries with multiple filters ... Use optimized specialized binary search for uint64 metricIDs instead of generic sort.Search.	2019-11-21 18:43:40 +02:00
Aliaksandr Valialkin	a02a57fbe9	lib/storage: verify the number of returned metricIDs in BenchmarkHeadPostingForMatchers	2019-11-20 15:40:03 +02:00
Aliaksandr Valialkin	3d1f4408cf	lib/decimal: increase decimal->float speed conversion for integer numbers	2019-11-20 14:09:10 +02:00
Aliaksandr Valialkin	f1f2eff08f	lib/decimal: reduce rounding error when converting from decimal to float with negative exponent ... While at it, slightly increase the conversion performance by moving fast path to the top of the loop.	2019-11-19 23:34:41 +02:00
Aliaksandr Valialkin	17eca31989	lib/backup: retrieve only the required metadata when reading GCS objects	2019-11-19 21:30:51 +02:00
Aliaksandr Valialkin	216a260ced	app/{vmbackup,vmrestore}: add -maxBytesPerSecond command-line flag for limiting the used network bandwidth during backup / restore	2019-11-19 20:32:43 +02:00
Aliaksandr Valialkin	9d1ee1e2ae	lib/backup: prevent from restoring to directory which is in use by VictoriaMetrics during the restore	2019-11-19 18:35:59 +02:00
Aliaksandr Valialkin	6ca4b94511	lib/storage: increase the number of created time series in BenchmarkHeadPostingForMatchers in order to be on par with Promethues ... The previous commit was accidentally creating 10x smaller number of time series than Prometheus and this led to invalid benchmark results. The updated benchmark results: benchmark old ns/op new ns/op delta BenchmarkHeadPostingForMatchers/n="1" 272756688 6194893 -97.73% BenchmarkHeadPostingForMatchers/n="1",j="foo" 138132923 10781372 -92.19% BenchmarkHeadPostingForMatchers/j="foo",n="1" 134723762 10632834 -92.11% BenchmarkHeadPostingForMatchers/n="1",j!="foo" 195823953 10679975 -94.55% BenchmarkHeadPostingForMatchers/i=~".*" 7962582919 100118510 -98.74% BenchmarkHeadPostingForMatchers/i=~".+" 7589543864 154955671 -97.96% BenchmarkHeadPostingForMatchers/i=~"" 1142371741 258003769 -77.42% BenchmarkHeadPostingForMatchers/i!="" 9964150263 159783895 -98.40% BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo" 216995884 10937895 -94.96% BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo" 202541348 10990027 -94.57% BenchmarkHeadPostingForMatchers/n="1",i!="" 486285711 87004349 -82.11% BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo" 350776931 53342793 -84.79% BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo" 380888565 54256156 -85.76% BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo" 89500296 21823279 -75.62% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo" 379529654 46671359 -87.70% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo" 424563825 53915842 -87.30% VictoriaMetrics uses 1GB of RAM during the benchmark (vs 3.5GB of RAM for Prometheus)	2019-11-18 19:48:27 +02:00
Aliaksandr Valialkin	6f61fd367a	lib/storage: add BenchmarkHeadPostingForMatchers similar to the benchmark from Prometheus ... See the corresponding benchmark in Prometheus - 23c0299d85/tsdb/head_bench_test.go (L52) The benchmark allows performing apples-to-apples comparison of time series search in Prometheus and VictoriaMetrics. The following article - https://www.robustperception.io/evaluating-performance-and-correctness - contains incorrect numbers for VictoriaMetrics, since there wasn't this benchmark yet. Fix it. Benchmarks can be repeated with the following commands from Prometheus and VictoriaMetrics source code roots: - Prometheus: GOMAXPROCS=1 go test ./tsdb/ -run=111 -bench=BenchmarkHeadPostingForMatchers - VictoriaMetrics: GOMAXPROCS=1 go test ./lib/storage/ -run=111 -bench=BenchmarkHeadPostingForMatchers Benchmark results: benchmark old ns/op new ns/op delta BenchmarkHeadPostingForMatchers/n="1" 272756688 364977 -99.87% BenchmarkHeadPostingForMatchers/n="1",j="foo" 138132923 1181636 -99.14% BenchmarkHeadPostingForMatchers/j="foo",n="1" 134723762 1141578 -99.15% BenchmarkHeadPostingForMatchers/n="1",j!="foo" 195823953 1148056 -99.41% BenchmarkHeadPostingForMatchers/i=~".*" 7962582919 8716755 -99.89% BenchmarkHeadPostingForMatchers/i=~".+" 7589543864 12096587 -99.84% BenchmarkHeadPostingForMatchers/i=~"" 1142371741 16164560 -98.59% BenchmarkHeadPostingForMatchers/i!="" 9964150263 12230021 -99.88% BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo" 216995884 1173476 -99.46% BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo" 202541348 1299743 -99.36% BenchmarkHeadPostingForMatchers/n="1",i!="" 486285711 11555193 -97.62% BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo" 350776931 5607506 -98.40% BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo" 380888565 6380335 -98.32% BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo" 89500296 2078970 -97.68% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo" 379529654 6561368 -98.27% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo" 424563825 6757132 -98.41% The first column (old) is for Prometheus, the second column (new) is for VictoriaMetrics. Prometheus was using 3.5GB of RAM during the benchmark, while VictoriaMetrics was using 400MB of RAM.	2019-11-18 18:47:02 +02:00
Aliaksandr Valialkin	d297b65089	lib/storage: add vm_cache_size_bytes{type="storage/hour_metric_ids"} metric	2019-11-13 20:26:05 +02:00
Aliaksandr Valialkin	494ad0fdb3	lib/storage: remove inmemory index for recent hour, since it uses too much memory ... Production workload shows that the index requires ~4Kb of RAM per active time series. This is too much for high number of active time series, so let's delete this index. Now the queries should fall back to the index for the current day instead of the index for the recent hour. The query performance for the current day index should be good enough given the 100M rows/sec scan speed per CPU core.	2019-11-13 18:08:58 +02:00
Aliaksandr Valialkin	633dd81bb5	lib/storage: add -disableRecentHourIndex flag for disabling inmemory index for recent hour ... This may be useful for saving RAM on high number of time series aka high cardinality	2019-11-13 15:10:12 +02:00
Aliaksandr Valialkin	f1620ba7c0	lib/storage: fix inmemory inverted index issues found in v1.29 ... Issues fixed: - Slow startup times. Now the index is loaded from cache during start. - High memory usage related to superflouos index copies every 10 seconds.	2019-11-13 13:35:38 +02:00
Aliaksandr Valialkin	87b39222be	Revert "lib/fs: do not postpone directory removal on NFS error" ... This reverts commit 21aeb02b46649ac9906cb37733f7b155a77a0db9.	2019-11-12 16:29:50 +02:00
Mike Poindexter	955a592106	Add test for invalid caching of tsids (#232) ... * Add test for invalid caching of tsids * Clean up error handling	2019-11-12 15:52:46 +02:00
Oleg Kovalov	74ba42d111	fix misspelled words (#229)	2019-11-12 00:18:24 +02:00
Aliaksandr Valialkin	c48e39eea9	lib/storage: add tests for dateMetricIDCache	2019-11-11 13:21:05 +02:00
Aliaksandr Valialkin	6bdde0d6d4	lib/storage: eliminate data race when updating lastSyncTime in dateMetricIDCache.Has	2019-11-10 22:04:23 +02:00
Aliaksandr Valialkin	5f52eb7653	lib/fs: do not postpone directory removal on NFS error ... Continue trying to remove NFS directory on temporary errors for up to a minute. The previous async removal process breaks in the following case during VictoriaMetrics start - VictoriaMetrics opens index, finds incomplete merge transactions and starts replaying them. - The transaction instructs removing old directories for parts, which were already merged into bigger part. - VictoriaMetrics removes these directories, but their removal is delayed due to NFS errors. - VictoriaMetrics scans partition directory after all the incomplete merge transactions are finished and finds directories, which should be removed, but weren't still removed due to NFS errors. - VictoriaMetrics panics when it finds unexpected empty directory. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-11-10 13:27:16 +02:00
Aliaksandr Valialkin	9ea2bd822e	lib/storage: implement per-day inverted index	2019-11-10 00:20:32 +02:00
Aliaksandr Valialkin	dea2f3efed	lib/storage: use specialized cache for (date, metricID) entries ... This improves ingestion performance.	2019-11-09 23:09:18 +02:00
Aliaksandr Valialkin	9a43902bd8	lib/storage: remove unused code from getMetricIDsForTimeRange: it is expected that time range is always non-zero	2019-11-09 19:03:51 +02:00
Aliaksandr Valialkin	c16e17dede	lib/storage: properly set time range when deleting time series	2019-11-09 18:50:02 +02:00
Aliaksandr Valialkin	8126007c15	lib/storage: obtain all the time series ids from (tag->metricIDs) rows instead of (metricID->TSID) rows, since this much faster	2019-11-09 18:04:26 +02:00
Aliaksandr Valialkin	50773348d3	lib/storage: small code prettifying	2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin	44fa8226df	lib/uint64set: remove superflouos check for item existence before deleting it in Set.Subtract	2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin	0bc54c23ce	lib/storage: inmemoryInvertedIndex prettifying	2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin	46e67bb78c	lib/storage: export vm_new_timeseries_created_total metric for determining time series churn rate	2019-11-08 19:58:21 +02:00
Aliaksandr Valialkin	0063c857f5	lib/storage: add inmemory inverted index for the last hour ... It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.	2019-11-08 19:37:46 +02:00
Aliaksandr Valialkin	7d7fbf890e	app/{vmbackup,vmrestore}: add vmbackup and vmrestore tools for creating backups on s3 or gcs from instant snapshots ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38	2019-11-07 21:26:43 +02:00
Aliaksandr Valialkin	89c03a5464	lib/storage: populate partition names from both small and big directories ... Certain partition directories may be missing after restoring from backups if they had no data. Re-create such directories on start.	2019-11-06 19:50:21 +02:00
Aliaksandr Valialkin	1c777e0245	lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter ... The origin of the error has been detected and documented in the code, so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`, so it could be monitored and alerted on high error rates. Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`, so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.	2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin	c567a4353a	lib/storage: take into account the requested time range when caching TSIDs for the given tag filters	2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin	c6564c5d26	lib/storage: dump incorrectly sorted items on a single line; this should simplify error reporting	2019-11-05 18:41:50 +02:00
Aliaksandr Valialkin	a10c4cad85	lib/storage: return back finalPartsToMerge from 2 to 3 in order to prevent from excessive merges in old partitions	2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin	e5b1fa0c38	lib/storage: separate the max inverted index scan loops per metric into fast and slow loops ... Slow loops could require seeks and expensive regexp matching, while fast loops just scans all the metricIDs for the given `tag=value` prefix. So these operations must have separate max loops multiplier.	2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin	f93c4f2493	lib/storage: skip repeated useless work when intersection of metricIDs with the given filter is too expensive ... This should improve performance for query filters over big number of time series.	2019-11-05 14:35:55 +02:00
Aliaksandr Valialkin	f48e97263c	lib/storage: reduce the maximum inverted index scans before giving up to label filters matching by metric name ... The new value reduces the amount of wasted work during index scans over big number of time series.	2019-11-05 14:35:53 +02:00
Aliaksandr Valialkin	d2f688c550	lib/storage: try potentially faster tag filters at first, then apply slower tag filters ... The fastest tag filters are non-negative non-regexp, since they are the most specific. The slowest tag filters are negative regexp, since they require scanning all the entries for the given label.	2019-11-05 14:35:48 +02:00
Aliaksandr Valialkin	2a38d30f93	lib/storage: pass pointer to MetricName in Fatalf, so it is properly detected as an interface with String() method ... This fixes lint errors	2019-11-04 01:06:45 +02:00
Artem Navoiev	e05500cbd4	add unittests for bytesutil and storage (#221)	2019-11-04 00:57:24 +02:00
Aliaksandr Valialkin	f5fbc3ffd7	lib/{storage,uint64set}: add Set.Union() function and use it	2019-11-04 00:48:32 +02:00
Aliaksandr Valialkin	23e078261e	lib/storage: tune the returned value from adjustMaxMetricsAdaptive	2019-11-04 00:45:28 +02:00
Aliaksandr Valialkin	386c349c8c	lib/storage: remove interface conversion in hot path during block merging ... This should improve merge speed a bit for parts with big number of small blocks.	2019-11-03 12:33:48 +02:00
Aliaksandr Valialkin	26ffc77622	lib/{storage,mergeset}: create missing partition directories after restoring from backups ... Backup tools could skip empty directories. So re-create such directories on the first run.	2019-11-02 02:27:19 +02:00
Aliaksandr Valialkin	5d439cc6f2	lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones ... Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.	2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin	1037053fed	lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones ... Time series with such values are quite common in monitoring space, so it would be great to have benchmarks for them.	2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin	44fab198e2	lib/uint64set: return an emptry set instead of nil set from Set.Clone, since the caller may add data to the cloned set ... This fixes the following panic in v1.28.1: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x783a7e] goroutine 1155 [running]: github.com/VictoriaMetrics/VictoriaMetrics/lib/uint64set.(*Set).Add(0x0, 0x15b3bfb41e8b71ec) github.com/VictoriaMetrics/VictoriaMetrics@/lib/uint64set/uint64set.go:57 +0x2e github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForRecentHours(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0x4a2f51, 0xaa01000000000000) github.com/VictoriaMetrics/VictoriaMetrics@/lib/storage/index_db.go:1951 +0x260 github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForTimeRange(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0xb296c0, 0xc00009cd80, 0x9bc640)	2019-11-01 16:12:21 +02:00
Aliaksandr Valialkin	6ab9c98a1e	app/vmstorage: add -bigMergeConcurrency and -smallMergeConcurrency flags for tuning the maximum number of CPU cores used during merges	2019-10-31 16:17:29 +02:00
Aliaksandr Valialkin	6a22727676	lib/storage: optimize getMetricIDsForRecentHours for per-tenant lookups	2019-10-31 15:51:09 +02:00
Aliaksandr Valialkin	ca480915ca	lib/storage: small cleanup in Storage.add	2019-10-31 14:30:22 +02:00
Aliaksandr Valialkin	22030b558f	lib/decimal: speed up FromFloat for common case with integers	2019-10-31 13:25:09 +02:00
Aliaksandr Valialkin	6510258a80	lib/decimal: increase float64->decimal conversion precision a bit ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213	2019-10-30 02:04:28 +02:00
Aliaksandr Valialkin	a27e034a40	lib/storage: get parts to merge after applying the limit on the number of concurrent merges ... This should reduce write amplification under high ingestion rate.	2019-10-30 00:09:44 +02:00
Aliaksandr Valialkin	5d2276dbf7	lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge	2019-10-29 12:45:37 +02:00
Aliaksandr Valialkin	f581b2736a	lib/fs: typo fix in comment to WriteFileAtomically	2019-10-29 11:31:34 +02:00
Aliaksandr Valialkin	eb513e7ba3	lib/decimal: increase float->decimal conversion precision for big numbers ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213	2019-10-28 13:23:54 +02:00
hanzai	52778da1f3	warns during rows addition (#214)	2019-10-20 23:38:51 +03:00
Aliaksandr Valialkin	78fc35c9b1	all: make fmt	2019-10-17 20:05:12 +03:00
Aliaksandr Valialkin	5b01b7fb01	all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin	12c8afc3f2	lib/memory: properly handle int overflow in sysTotalMemory ... This should fix builds on 32-bit architectures such as arm. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin	dc2f822577	lib/prompb: removed outdated README.md	2019-10-14 22:16:36 +03:00
Aliaksandr Valialkin	661b8ede5b	lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order	2019-10-09 12:13:43 +03:00
Aliaksandr Valialkin	7f4a04ee6a	lib/storage: typo fix in comment to maxRowsPerSmallPart.	2019-10-08 18:51:56 +03:00
Aliaksandr Valialkin	7e410e1412	lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering. ... This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.	2019-10-08 16:35:39 +03:00
Aliaksandr Valialkin	9fce611fbb	lib/mergeset: reduce the maximum number of cached blocks, since there are reports on OOMs due to too big caches ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/195	2019-09-30 12:27:30 +03:00
Aliaksandr Valialkin	d0b4590099	lib/storage: optimize TSID comparison	2019-09-26 14:20:02 +03:00
Aliaksandr Valialkin	95e3d648cb	lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows ... This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.	2019-09-26 13:13:58 +03:00
Aliaksandr Valialkin	2b8358726f	lib/storage: properly match labels against regexp with (?i) flag ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161	2019-09-26 11:03:26 +03:00
Aliaksandr Valialkin	4e3871ac1e	lib/storage: add missing break in removeDuplicateMetricIDs	2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin	4468f9f966	lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index	2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin	adc18c3ee6	lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items ... This should reduce the amounts of used RAM during queries with filters over big number of time series.	2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin	c64fb91a43	lib/uint64set: optimize Set.AppendTo	2019-09-25 00:34:31 +03:00
Aliaksandr Valialkin	de0e4eee2c	lib/storage: create and use lib/uint64set instead of map[uint64]struct{} ... This should improve inverted index search performance for filters matching big number of time series, since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls. See the corresponding benchmarks in `lib/uint64set`.	2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin	2212d0e421	lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows	2019-09-24 19:32:58 +03:00
Aliaksandr Valialkin	9307de1b92	lib/storage: limit the number of metricIDs in tag->metricIDs row ... This reduces the overhead on index and metaindex in lib/mergeset	2019-09-24 00:50:47 +03:00
Aliaksandr Valialkin	7734fc8012	lib/storage: share tsids across all the partSearch instances ... This should reduce memory usage when big number of time series matches the given query.	2019-09-23 22:36:16 +03:00
Aliaksandr Valialkin	67a2bcb98a	lib/{storage,mergeset}: verify PrepareBlock callback results ... Do not touch the first and the last item passed to PrepareBlock in order to preserve sort order of mergeset blocks.	2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin	3304dc1e85	lib/mergeset: detect whether we are in test by executable suffix	2019-09-22 23:12:35 +03:00
Aliaksandr Valialkin	d2ed8cb0b2	lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID ... The first item from each mergeset block goes into index (lib/mergeset.blockHeader), so it must be short in order to reduce index size.	2019-09-22 19:37:50 +03:00
Aliaksandr Valialkin	0a9cb6368e	lib/workingsetcache: remove data race when resetting c.misses	2019-09-22 19:37:09 +03:00
Aliaksandr Valialkin	7d13c31566	lib/{storage,mergeset}: merge tag->metricID rows into tag->metricIDs rows for common tag values ... This should improve lookup performance if the same `label=value` pair exists in big number of time series. This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows occupy less space than the original `tag->metricID` rows.	2019-09-20 22:06:23 +03:00
Aliaksandr Valialkin	272e2f77c9	lib/encoding: optimize UnmarshalUint* and UnmarshalInt*	2019-09-20 13:08:24 +03:00
Aliaksandr Valialkin	7e0c6d4ca6	lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries ... The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries and MetricID->TSID entries are usually shorter than tag->MetricID entries. This should improve performance when selecting all the metricIDs.	2019-09-20 11:57:57 +03:00
Aliaksandr Valialkin	bf8505353a	lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts	2019-09-19 21:48:36 +03:00
Aliaksandr Valialkin	ebbef20535	lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations	2019-09-19 20:13:54 +03:00
Aliaksandr Valialkin	89234f395d	lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs	2019-09-19 20:08:13 +03:00
Aliaksandr Valialkin	6e586fa09c	lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects	2019-09-19 17:51:10 +03:00
Aliaksandr Valialkin	410f993bf6	lib/mergeset: fill partHeader.firstItem on first block flush	2019-09-19 17:48:22 +03:00
Aliaksandr Valialkin	c05885fb5f	lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless ... This will save CPU on subsequent calls for this filter	2019-09-18 19:11:44 +03:00
Aliaksandr Valialkin	db71c940ea	lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there	2019-09-17 23:18:37 +03:00
Aliaksandr Valialkin	41ef6b060e	lib/mergeset: properly check for sorted block headers ... Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181	2019-09-13 21:59:38 +03:00
hanzai	08cde5e3f4	lib/workingsetcache: adjust switching from mode=split to mode=whole smoothly and load cachefile successfully	2019-09-13 19:13:16 +03:00
Aliaksandr Valialkin	b101064f8b	all: report the number of bytes read on io.ReadFull error ... This should simplify error investigation similar to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175	2019-09-11 14:50:24 +03:00
Aliaksandr Valialkin	568ff61dcf	lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache	2019-09-09 11:42:45 +03:00
Aliaksandr Valialkin	2c654258ef	lib/fs: add MustStopDirRemover for waiting until pending directories are removed on graceful shutdown ... This patch is mainly required for laggy NFS. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-09-05 11:17:17 +03:00
Aliaksandr Valialkin	2c2bd897dd	lib/storage: remove duplicate tag keys on MetricName.Marshal call ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172	2019-09-04 18:13:51 +03:00
Aliaksandr Valialkin	82bfe818d0	lib/fs: try harder with directory removal on NFS in the event of temporary lock ... Do not give up after 11 attempts of directory removal on laggy NFS. Add `vm_nfs_dir_remove_failed_attempts_total` metric for counting the number of failed attempts on directory removal. Log failed attempts on directory removal after long sleep times. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-09-04 12:24:41 +03:00
Aliaksandr Valialkin	0b0153ba3d	lib/storage: invalidate tagFilters -> TSIDS cache when newly added index data becomes visible to search ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163	2019-08-29 15:08:44 +03:00
Aliaksandr Valialkin	8504a38214	lib/prombp: apply ba06b47c16 ... The following commands used: gofmt -r '(uint64(x)&0x7F)<<shift -> uint64(x&0x7F)<<shift' -w ./lib/prompb/ gofmt -r '(int64(x)&0x7F)<<shift -> int64(x&0x7F)<<shift' -w ./lib/prompb/	2019-08-29 13:35:54 +03:00
Aliaksandr Valialkin	604a4312f9	all: port to FreeBSD on GOARCH=amd64	2019-08-28 01:46:09 +03:00
Aliaksandr Valialkin	da07a6fb38	lib/storage: go fmt	2019-08-27 14:28:24 +03:00
Aliaksandr Valialkin	a63b69e9e2	lib/storage: report proper maxMetrics limit when more than -search.maxUniqueTimeseries series match the given filters	2019-08-27 14:21:31 +03:00
Aliaksandr Valialkin	82e813bad3	lib/storage: properly handle (?i) in the tag filter regexp ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161	2019-08-26 00:44:56 +03:00
Aliaksandr Valialkin	e2eac858b5	lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/159 This simplifies error detection additionally to the `vm_rows_ignored_total` counters.	2019-08-25 15:29:09 +03:00
Aliaksandr Valialkin	0a8dd9cc9a	lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent ... This should improve query speed over recent data on machines with big amounts of RAM	2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin	bc576fb386	lib/storage: properly limit the number of output rows in small and big parts storage ... Previously small parts storage didn't take into account the available disk space for big parts.	2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin	947decb3dd	lib/storage: remove outdated comment on maxRowsPerSmallPart ... The commend became outdated after the commit ed6ac1a5df027f0dfc22448e3b27c26b6f77c67a, which stops merging of small parts on graceful shutdown instead of waiting for their completion.	2019-08-25 13:46:10 +03:00
Aliaksandr Valialkin	c197641978	all: return 503 http error if service is temporarily unavailable ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156	2019-08-23 09:49:50 +03:00
Aliaksandr Valialkin	e734076f0f	app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries	2019-08-23 08:47:18 +03:00
Aliaksandr Valialkin	4ed63d033a	lib/storage: add benchmarks for regexp filter match / mismatch ... These benchmarks allow estimate the performance of regexp filters in promql	2019-08-22 16:37:19 +03:00
Aliaksandr Valialkin	6ec6a8d7c1	lib/storage: try slower path for searching the tag filter with the minimum number of matching time series before giving up with increase -search.maxUniqueTimeseries error	2019-08-19 16:07:05 +03:00
Aliaksandr Valialkin	c59f5c4865	lib/storage: pre-allocate memory for blockHeader slice in unmarshalBlockHeaders ... This reduces memory usage and memory fragmentation when working with big number of time series	2019-08-19 12:46:45 +03:00
Aliaksandr Valialkin	f696cc503a	lib/workingsetcache: automatically detect when it is better to double cache capacity	2019-08-15 22:58:04 +03:00
Aliaksandr Valialkin	99eed2ca14	lib/storage: properly cache tagFilters -> TSIDs entries from historical index	2019-08-14 02:32:25 +03:00
Aliaksandr Valialkin	f1d81b9405	lib/storage: compress contents of cache for tagFilters -> TSIDs ... This should increase cache capacity	2019-08-14 02:32:22 +03:00
Aliaksandr Valialkin	8c2158af24	all: use workingsetcache instead of fastcache ... This should reduce the amount of RAM required for processing time series with non-zero churn rate. The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.	2019-08-13 21:40:28 +03:00
Aliaksandr Valialkin	51263b1a45	lib/fs: add test for IsTemporaryFileName	2019-08-13 21:33:54 +03:00
Aliaksandr Valialkin	5a7ab0d90b	lib/storage: remove broken BenchmarkIndexDBSearchTSIDs	2019-08-13 20:21:23 +03:00
Aliaksandr Valialkin	39f3f3a517	lib: move common code for creating flock.lock file into fs.CreateFlockFile	2019-08-13 01:46:20 +03:00
Aliaksandr Valialkin	73f866d874	lib/fs: atomically create file with the given contents on WriteFileAtomically ... This should prevent from `transaction` and `metadata.json` files corruption on unclean shutdown such as OOM, `kill -9`, power loss, etc. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/148	2019-08-12 15:02:04 +03:00
Aliaksandr Valialkin	4fb635b0c9	lib/storage: do not change timestamps to constant rate if values are constant or have constant delta ... This breaks the original timestamps, which results in issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/120 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/141 .	2019-08-06 15:40:17 +03:00
Aliaksandr Valialkin	f56c1298ad	app/vmstorage: add vm_concurrent_addrows_* metrics for tracking concurrency for Storage.AddRows calls ... Track also the number of dropped rows due to the exceeded timeout on concurrency limit for Storage.AddRows. This number is tracked in `vm_concurrent_addrows_dropped_rows_total`	2019-08-06 15:08:43 +03:00
Aliaksandr Valialkin	a3ecf3c1f7	lib/storage: properly reset partSearch.fetchData in partSearch.reset	2019-08-05 09:55:50 +03:00
Aliaksandr Valialkin	880b1d80b1	app/vmselect: optimize /api/v1/series by skipping storage data ... Fetch and process only time series metainfo.	2019-08-04 23:00:46 +03:00
Aliaksandr Valialkin	a99e89945e	lib/decimal: modernize tests a bit	2019-07-31 21:09:54 +03:00
Aliaksandr Valialkin	b7c4b0c6d2	lib/storage: fix matching against tag filter with empty name ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/137	2019-07-30 15:15:21 +03:00
Aliaksandr Valialkin	c6bec48927	lib/storage: add metrics for calculating skipped rows outside the retention ... The metrics are: - vm_too_big_timestamp_rows_total - vm_too_small_timestamp_rows_total	2019-07-26 14:11:56 +03:00
Aliaksandr Valialkin	f2e8d54fb0	lib/encoding/zstd: go fmt	2019-07-25 01:37:57 +03:00
Aliaksandr Valialkin	97b5dc7122	lib/encoding/zstd: disable CRC checks in pure Go build ... This should give slightly better compression and decompressions performance. Additionally this shaves off 4 bytes per each compressed block.	2019-07-24 19:17:32 +03:00
Aliaksandr Valialkin	54f035d4ce	all: small updates after PR #114	2019-07-24 17:43:43 +03:00
Aliaksandr Valialkin	7a133567fb	lib/encoding: small fixes in tests after the PR #114	2019-07-24 17:43:39 +03:00
Roman Khavronenko	fcf09aaa3c	all: add Pure Go build (pull request #114) ... Updates #94	2019-07-24 17:43:32 +03:00
Aliaksandr Valialkin	3fae34eeb4	lib/encoding: improve gauge series detection ... - Series with negative values are always gauges - Counters may only have increasing values with possible counter resets This should improve compression ratio for gauge series which were previously mistakenly detected as counters.	2019-07-20 14:05:25 +03:00
Aliaksandr Valialkin	7abb96b454	lib/netutil: do not count timeouts as network errors	2019-07-15 23:06:13 +03:00
Aliaksandr Valialkin	73a47d2a53	lib/storage: remove unused function isTooBigTimeRangeForDateMetricIDs	2019-07-12 02:28:40 +03:00
Aliaksandr Valialkin	97f9397687	lib/storage: do not reduce maxMetrics on time ranges exceeding maxDaysForDateMetricIDs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/95	2019-07-12 02:21:52 +03:00
Aliaksandr Valialkin	4ca66344ee	lib/storage: do not pollute inverted index with data for samples outside the retention period	2019-07-11 17:11:33 +03:00
Aliaksandr Valialkin	0522efb2d6	lib/storage: add missing tagFilter.Marshal func	2019-07-11 15:01:01 +03:00
Aliaksandr Valialkin	12b1d67b41	lib/storage: use fast path for orSuffix when searching for metricIDs against plain tag value	2019-07-11 14:48:51 +03:00
Aliaksandr Valialkin	bf2e1b0ac1	lib/storage: remember and skip individual tag filters matching too many metrics ... This saves CPU time by skipping useless matching for individual tag filters.	2019-07-11 14:48:47 +03:00
Aliaksandr Valialkin	ba8195c58e	all: consistency renaming: bytesSize -> sizeBytes	2019-07-10 00:47:42 +03:00
Aliaksandr Valialkin	41f512af1c	all: add vm_data_size_bytes metrics for easy monitoring of on-disk data size and on-disk inverted index size	2019-07-04 19:43:04 +03:00
Aliaksandr Valialkin	6682a35731	lib/memory: attempt #3 to determine memory limit for LXC container ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84	2019-07-01 14:01:57 +03:00
Aliaksandr Valialkin	b6ea1a7d5e	lib/mergeset: make fmt	2019-06-29 14:25:46 +03:00
Aliaksandr Valialkin	ffc1bb00f6	lib/storage: skip non-matching metricIDs in sortedFilter ... This should improve performance for big sorteFilter lists.	2019-06-29 13:49:40 +03:00
Aliaksandr Valialkin	2257dcd278	lib/mergeset: speed up binarySearchKey by skipping the first item during binary search	2019-06-29 13:49:32 +03:00
Aliaksandr Valialkin	6ea12a079e	lib/memory: remove TestReadLXCMemoryLimit, since it doesnt work in Travis	2019-06-28 18:23:06 +03:00
Aliaksandr Valialkin	d0732d3137	lib/memory: attempt #2 to determine memory limit inside LXC container ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84	2019-06-28 18:08:57 +03:00
Aliaksandr Valialkin	ad436757c3	lib/memory: an attempt to read proper memory limit inside LXC container ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84	2019-06-28 15:35:02 +03:00
Aliaksandr Valialkin	416d27ef11	lib/storage: optimize time series search by regexp filter ... This should improve search speed on label filters like `{foo=~"bar.+baz"}`	2019-06-27 16:18:00 +03:00
Aliaksandr Valialkin	ee23a143b9	lib/storage: make sure non-nil args are passed to openIndexDB	2019-06-25 20:10:08 +03:00
Aliaksandr Valialkin	8b0a63722f	lib/storage: reduce too big maxMetrics in getTagFilterWithMinMetricIDsCountAdaptive ... This should improve performance on inverted index search for big amount of unique time series when big -search.maxUniqueTimeseries is set.	2019-06-25 19:57:31 +03:00
Aliaksandr Valialkin	0263cb0adc	lib/storage: free up memory from caches owned by indexDB when it is deleted	2019-06-25 14:41:16 +03:00
Aliaksandr Valialkin	362e187011	lib/storage: use unversioned keys for tag cache in extDB ... Data in ExtDB cannot be changed, so it is OK to use unversioned keys for tag cache. This should improve performance for index lookups over big amount of time series.	2019-06-25 13:15:42 +03:00
Aliaksandr Valialkin	51e2f3b48f	lib/storage: skip searching in extDB if it doesn't contain items for the given time range ... This should improve inverted index search performance for big amount of unique time series when the search is performed only on recent data.	2019-06-25 12:57:56 +03:00
Aliaksandr Valialkin	fb9358635d	lib/storage: mention source parts on merge error ... This should improve determining broken source part. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76	2019-06-24 14:09:46 +03:00
Aliaksandr Valialkin	9164c223ec	all: initial stubs for Windows support; see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2019-06-20 20:07:41 +03:00
Aliaksandr Valialkin	786beb8fc8	Makefile: enable golangci-lint in make check_all	2019-06-20 15:00:58 +03:00
Aliaksandr Valialkin	9cac11db64	lib/storage: typo fixes found by golangci-lint; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:38:45 +03:00
Aliaksandr Valialkin	7778030f9f	lib/netutil: remove unused TCPListener.name; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:36:19 +03:00
Aliaksandr Valialkin	a0c22a6830	app/vmstorage: add vm_cache_entries{type="storage/hour_metric_ids"} metric for tracking active time series count	2019-06-19 18:37:38 +03:00
Aliaksandr Valialkin	24ae3ef532	lib/prompb: remove superflouos bytes copying in ReadSnappy	2019-06-18 21:02:02 +03:00
Aliaksandr Valialkin	e40224d5de	lib/flagutil: add NewArray helper func	2019-06-18 10:44:09 +03:00
Aliaksandr Valialkin	f9e1d32168	lib/storage: persist metric ids for the current and the previous hour on graceful shutdown ... This should improve performance after restart when the db contains a lot of time series with high time series churn (i.e. metrics from Kubernetes with many pods and frequent deployments)	2019-06-14 07:55:09 +03:00
Aliaksandr Valialkin	b0b93e3d50	lib/fs: sync parent dir in MustRemoveAll only if it exists ... The parent directory may be non-existing when the deleted directory didn't exist before the MustRemoveAll call	2019-06-12 02:16:15 +03:00
Aliaksandr Valialkin	18d6f293f7	lib/fs: consolidate *RemoveAll* funcs into a single MustRemoveAll func ... The func syncs parent dir in order to persist directory removal in the event of power loss	2019-06-12 01:55:18 +03:00
Aliaksandr Valialkin	28d9904efc	lib/fs: panic with fatal error when directories cannot be removed ... Unremoved directories may lead to inconsistent data directory, so VictoriaMetrics will fail to start next time. So panic on the first error when trying to remove directory in order to simplify recover process.	2019-06-12 01:20:10 +03:00
Aliaksandr Valialkin	d897bc3f08	lib/fs: attempt #2 to work around NFS issue with directory removal ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61	2019-06-12 01:07:29 +03:00
Aliaksandr Valialkin	51e2e255a6	lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error	2019-06-11 23:13:45 +03:00
Aliaksandr Valialkin	3fa4c28f6b	lib/fs: make sure the created directory remains visible in the fs in the event of power loss ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63	2019-06-11 23:08:17 +03:00
Aliaksandr Valialkin	0b7f751f60	lib/fs: use filepath.Dir instead of filepath.Split, since the filename is unused	2019-06-11 22:54:23 +03:00
Aliaksandr Valialkin	b491045a4b	lib/{storage,mergeset}: sync filenames inside part when finalizing the part ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63	2019-06-11 21:51:19 +03:00
Aliaksandr Valialkin	3437c30180	all: try hard removing directory with contents ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61	2019-06-11 01:58:08 +03:00
Aliaksandr Valialkin	75a0acf72d	app/vmselect: add /api/v1/labels/count handler for quick detection of labels with the maximum number of distinct values	2019-06-10 19:54:55 +03:00
Aliaksandr Valialkin	0ccedbdfd2	lib/storage: mention the accountID and projectID in error message when filtering out other (accountID, projectID) entries	2019-06-10 14:43:53 +03:00
Aliaksandr Valialkin	d54f5fec0b	lib/storage: skip adaptive searching for tag filter matching the minimum number of metrics if the identical previous search didn't found such filter ... This should improve speed for searching metrics among high number of time series with high churn rate like in big Kubernetes clusters with frequent deployments.	2019-06-10 14:07:47 +03:00
Aliaksandr Valialkin	27e50e86f4	lib/storage: factor out getTagFilterWithMinMetricIDsCountAdaptive from updateMetricIDsForTagFilters	2019-06-10 13:26:00 +03:00
Aliaksandr Valialkin	b69d3dbd0c	lib/storage: filter out metricIDs from another (AccountID, ProjectID) in getMetricIDsForRecentHours	2019-06-10 13:05:16 +03:00
Aliaksandr Valialkin	3059ae7be0	lib/storage: give clearer names to more functions	2019-06-10 12:59:33 +03:00
Aliaksandr Valialkin	d3a024d2d6	lib/storage: give more clear names to functions	2019-06-10 12:50:22 +03:00
Aliaksandr Valialkin	00e0760608	lib/storage: test GetSeriesCount	2019-06-10 12:40:33 +03:00
Aliaksandr Valialkin	e4cba5a7ed	lib/storage: make getSeriesCount func indexSearch method	2019-06-10 12:29:24 +03:00
Aliaksandr Valialkin	d882afa905	lib/storage: optimize time series lookup for recent hours when the db contains many millions of time series with high churn rate (aka frequent deployments in Kubernetes)	2019-06-09 19:14:04 +03:00
Aliaksandr Valialkin	8cf0a0e59c	app/vminsert: split vm_rows_inserted_total into per-(accountID, projectID) metrics ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/59	2019-06-07 22:11:20 +03:00
Aliaksandr Valialkin	1d87abc8eb	lib/procutil: typo fix in comment to WaitForSigterm	2019-06-03 16:54:37 +03:00
Aliaksandr Valialkin	a2986cde70	lib/storage: tune updating a map with today`s metric ids ... - Increase update iterval from 1s to 10s. This should reduce CPU usage for large amounts of metric ids with constant churn. - Reduce pendingTodayMetricIDsLock lock duration during the update.	2019-06-02 22:00:13 +03:00
Aliaksandr Valialkin	e27fd5148a	lib/storage: speed up checking metricID existence in the list for the current date	2019-06-02 18:34:20 +03:00
Aliaksandr Valialkin	a6d02ff275	lib/timerpool: use timer pool in concurrency limiters ... This should reduce the number of memory allocations in highly loaded system	2019-05-28 17:30:10 +03:00
Aliaksandr Valialkin	bdf696ef18	all: fix misspellings	2019-05-25 21:51:24 +03:00
Aliaksandr Valialkin	121a920a18	Makefile: add -s flag to go fmt in make fmt command	2019-05-25 21:44:36 +03:00
Aliaksandr Valialkin	9b64dfee4b	lib/encoding: add vm_zstd_block_{compress|decompress}_calls_total for determining the number CompressZSTD / DecompressZSTD calls	2019-05-24 13:01:15 +03:00
Aliaksandr Valialkin	b23352dc9e	lib/encoding: add vm_zstd_block_{original|compressed}_bytes_total metrics for rough estimation of block compression ratio	2019-05-24 12:34:51 +03:00
Aliaksandr Valialkin	f67f40d63a	lib/encoding: substitute CompressZSTD with CompressZSTDLevel	2019-05-24 12:32:49 +03:00
Aliaksandr Valialkin	a26e774eca	lib/httpserver: add -http.disableResponseCompression flag, which may help saving CPU resources at the cost of higher network bandwidth usage	2019-05-24 12:20:36 +03:00
Aliaksandr Valialkin	820cdae88d	lib/decimal: add a comment explaining weird code in maxUpExponent. Fixes #29	2019-05-23 17:18:50 +03:00
Aliaksandr Valialkin	24578b4bb1	all: open-sourcing cluster version	2019-05-23 00:25:38 +03:00
Aliaksandr Valialkin	1836c415e6	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00