Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-16 00:41:24 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
1,034 Commits 114 Branches 748 Tags 509 MiB
d99f48aa48
Commit Graph

403 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
7fbfef2aee lib/promscrape: extract common auth code to lib/promauth 2020-04-13 12:59:22 +03:00
Aliaksandr Valialkin
e0c6da8e2a lib/storage: disable deduplication after dedup tests are complete 
The rest of tests expect that the de-duplication is disabled.
 2020-04-10 17:33:38 +03:00
Aliaksandr Valialkin
8ed0d5471a lib/storage: correctly handle -dedup.minScrapeInterval values smaller than 8ms 
Such small values may be used for removing samples with duplicate timestamps.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/409 for details.
 2020-04-10 16:40:41 +03:00
Aliaksandr Valialkin
0b2f678d8e lib/{storage,mergeset}: make sure that requests and misses cache counters never go down 2020-04-10 14:44:52 +03:00
Aliaksandr Valialkin
661cfb03e2 lib/protoparser: add -*TrimTimstamp command-line flags for Influx, Graphite, OpenTSDB and CSV data 
These flags can be used for reducing disk space usage for timestamps data ingested over the given protocols
 2020-04-10 12:44:46 +03:00
Aliaksandr Valialkin
f0b08dbd9e lib/workingsetcache: accumulate stat counters on cache rotation 
This should prevent from cache stats counters going down after cache rotation,
which may corrupt `cache hit ratio` graph on the official Grafan dasbhoards
when using the following query:

    1 - (sum(rate(vm_cache_misses_total[5m])) by (type) / sum(rate(vm_cache_requests_total[5m])) by (type))
 2020-04-10 11:51:47 +03:00
Aliaksandr Valialkin
28c65b58a2 lib/memory: add more details to -memory.allowedPercent help message 2020-04-09 15:34:21 +03:00
Aliaksandr Valialkin
84fa146792 lib/httpserver: remove unnecessary http.HandlerFunc wrapper in gzipHandler 2020-04-01 18:14:47 +03:00
Aliaksandr Valialkin
0ad7aaf535 lib/storage: add missing reset for tagFilter.matchesEmptyValue on tagFilter.Init 2020-04-01 17:40:27 +03:00
Aliaksandr Valialkin
c189104be7 lib/promscrape: reduce timestamp jitter when scraping targets 
This should improve compression for timestamps
 2020-04-01 16:13:01 +03:00
Aliaksandr Valialkin
4c56acbafa lib/storage: remove duplicate data points on 7/8*minScrapeInterval interval instead of 1/2*minScrapeInterval 
This should reduce storage usage and should improve deduplication accuracy
 2020-04-01 15:47:04 +03:00
Aliaksandr Valialkin
504ea876f2 lib/storage: handle errors returned from TagFilters.Add when cloning TagFilters with negative filter 2020-03-31 16:18:34 +03:00
Aliaksandr Valialkin
ef714e01c1 lib/storage: add fast path for the previous indexdb search if it doesn't contain per-day inverted index yet 2020-03-31 12:35:15 +03:00
Aliaksandr Valialkin
7e755b4bac lib/storage: optimize per-day inverted index search for tag filters matching big number of time series 
- Sort tag filters in the ascending number of matching time series
  in order to apply the most specific filters first.
- Fall back to metricName search for filters matching big number of time series
  (usually this are negative filters or regexp filters).
 2020-03-31 00:53:29 +03:00
Aliaksandr Valialkin
d450249955 lib/storage: properly handle {label=~"foo|"} filters as Prometheus does 
Such filters must match all the time series with `label="foo"` plus all the time series without `label`

Previously only time series with `label="foo"` were matched.
 2020-03-30 20:21:47 +03:00
Aliaksandr Valialkin
b47444e69d lib/envflag: add -envflag.prefix for setting optional prefix for environment vars 2020-03-30 15:51:44 +03:00
kreedom
f058efb3d1 [vmalert] config parser (#393) 
* [vmalert] config parser

* make linter be happy

* fix test

* fix sprintf add test for rule validation
 2020-03-29 01:49:40 +02:00
Aliaksandr Valialkin
42c290ce9f lib/httpserver: add -http.maxGracefulShutdownDuration command-line flag for tuning the maximum duration required for graceful shutdown of http server 2020-03-27 20:10:05 +02:00
Aliaksandr Valialkin
8fa80a2dbc lib/uint64set: remove zero buckets after Set.Intersect 2020-03-27 01:16:34 +02:00
Aliaksandr Valialkin
7a35447031 lib/uint64set: small code cleanup and perf tuning 
* Remember the last accessed bucket on Has() call.
* Inline fast paths inside Add() and Has() calls.
* Remove fragile code with maxUnsortedBuckets inside bucket32.
 2020-03-25 15:29:59 +02:00
Aliaksandr Valialkin
cce936de5b lib/uint64set: go fmt 2020-03-24 22:28:09 +02:00
Aliaksandr Valialkin
7cdac6634c lib/storage: serialize snapshot creation process with mutex 
This guarantees that the snapshot contains all the recently added data
from inmemory buffers when multiple concurrent calls to Storage.CreateSnapshot are performed.
 2020-03-24 22:27:28 +02:00
Aliaksandr Valialkin
c31b956355 lib/uint64set: added more tests 2020-03-24 22:27:26 +02:00
Aliaksandr Valialkin
31a533656e lib/storage: remove obsolete code 2020-03-13 22:42:42 +02:00
Aliaksandr Valialkin
bf1869d33d lib/promscrape: allow overriding external_labels as Prometheus does 
Prometheus docs at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config say:

> In communication with external systems, they are always applied only
> when a time series does not have a given label yet and are ignored otherwise.

Though this may result in consistency chaos when scrape targets override `external_labels`,
let's stick with Prometheus behavior for the sake of backwards compatibility.

There is last resort in vmagent with `-remoteWrite.label`, which consistently
sets the configured labels to all the metrics before sending them to remote storage.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/366
 2020-03-12 20:25:27 +02:00
Aliaksandr Valialkin
0e7a71a245 app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369
 2020-03-12 19:13:56 +02:00
Aliaksandr Valialkin
fa7910fba1 lib/protoparser/csvimport: add missing metric vm_rows_invalid_total{type="csvimport"} 2020-03-12 15:28:10 +02:00
Aliaksandr Valialkin
375d5483fa lib/promscrape: remove possible races when registering and de-registering scrape workers for /targets page 2020-03-11 16:30:43 +02:00
Aliaksandr Valialkin
187fd89c70 lib/promscrape: consistently update /targets page after SIGHUP 2020-03-11 03:20:38 +02:00
Aliaksandr Valialkin
2f0a36044c app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv 2020-03-10 21:17:40 +02:00
Aliaksandr Valialkin
7545784a49 all: fix golangci-lint issues 2020-03-10 19:40:03 +02:00
Aliaksandr Valialkin
d39dd8aa69 lib/promscrape: do not retry idempotent requests when scraping targets 
This should prevent from the following unexpected side-effects of idempotent request retries:
- increased actual timeout when scraping the target comparing to the configured scrape_timeout
- increased load on the target

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/357
 2020-03-09 13:31:20 +02:00
Aliaksandr Valialkin
12789a4621 app/vmagent: do not allow non-supported fields in -remoteWrite.relabelConfig and file_sd_configs 
This should reduce possible confusion like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/363
 2020-03-06 20:19:34 +02:00
Aliaksandr Valialkin
0d893eff36 Makefile: add build and test rules with enabled race detector. These rules have -race suffix 
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
 2020-03-05 12:05:16 +02:00
Aliaksandr Valialkin
31a76a7b3a lib/promscrape: consistency renaming: stopCh -> globalStopCh 2020-03-03 20:08:22 +02:00
Aliaksandr Valialkin
f01d1bf4a8 app/vmagent: add -remoteWrite.maxDiskUsagePerURL for limiting the maximum disk usage for each -remoteWrite.url buffer 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352
 2020-03-03 19:49:20 +02:00
Aliaksandr Valialkin
c3b239eb1a lib/protoparser/prometheus: allow trailing comma in tags list 
The trailing comma is generated by cloudwatch exporter.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/350
 2020-03-02 22:23:28 +02:00
Aliaksandr Valialkin
6a1aab88fd lib/protoparser: metrics renaming: vm_protoparser_<type>_* -> vm_protoparser_*{type="<type>"} 
This should improve composability of these metrics in PromQL queries
 2020-02-28 20:19:59 +02:00
Aliaksandr Valialkin
c2e602286c lib/persistentqueue: reset chunk file when the persistent queue is empty 2020-02-28 20:06:59 +02:00
Aliaksandr Valialkin
cf9aee4ec3 all: properly split vm_deduplicated_samples_total among cluster components 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345
 2020-02-27 23:47:51 +02:00
Aliaksandr Valialkin
5e7b4795bd lib/envflag: typo fix in docs to -envflag.enable: envoronment->environment 2020-02-27 21:56:46 +02:00
Aliaksandr Valialkin
e6a481ab11 lib/promscrape: properly reload new configs on SIGHUP 2020-02-26 13:54:24 +02:00
Aliaksandr Valialkin
fa6815712f lib/promscrape: go fmt 2020-02-26 13:24:40 +02:00
Aliaksandr Valialkin
f2a6948a14 lib/promscrape: do not add missing port to __address__ label in order to be consistent with Prometheus behavior 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/331
 2020-02-25 20:50:18 +02:00
Aliaksandr Valialkin
c4194020ef app/vmagent: do not allow sending unpacked requests with sizes exceeding -maxInsertRequestSize 2020-02-25 19:35:43 +02:00
Aliaksandr Valialkin
2471340e0d app/vmagent: add ability to accept Influx line protocol data via TCP and UDP 
Just set `-influxListenAddr` command-line flag

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333
 2020-02-25 19:18:01 +02:00
Aliaksandr Valialkin
7a045125cc lib/fs: typo fix: read blocks bigger than 8KB via pread() call instead of using mmap 2020-02-25 18:04:06 +02:00
Aliaksandr Valialkin
13ee8271d0 lib/envflag: substitute dots with underscores in env var names if -envflag.enable is set 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
 2020-02-24 21:15:11 +02:00
Aliaksandr Valialkin
7ee7614e90 app/vmagent: initial implementation for vmagent 2020-02-23 17:31:54 +02:00
Aliaksandr Valialkin
110cce24d9 lib/storage: add vm_ prefix to deduplicated_samples_total metric 2020-02-21 19:33:36 +02:00
Aliaksandr Valialkin
9d279e26a7 lib/protoparser/prometheus: skip leading whitespace from tag names 2020-02-16 19:06:23 +02:00
Aliaksandr Valialkin
a2b81b71b9 lib/storage: typo fix 2020-02-16 15:53:48 +02:00
Aliaksandr Valialkin
ad4cb9f3ca lib/storage: prevent from clobbering nin-nil lastError in Storage.add 2020-02-16 15:51:35 +02:00
Aliaksandr Valialkin
846d7fa7e9 app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions 
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
 2020-02-13 17:01:50 +02:00
Aliaksandr Valialkin
e3b18ca1ab lib/mergeset: skip createing temporary part objects when merging source inmemory parts 
This should reduce CPU usage when adding new entries to inverted index.
This should alos prevent from creating stalled cleaner goroutines for the created temporary parts,
since they were never closed.

This should fix the following issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .
 2020-02-13 14:09:13 +02:00
Aliaksandr Valialkin
347aaba79d lib/{storage,mergeset}: use time.Ticker instead of time.Timer where appropriate 
It has been appeared that time.Timer was used in places where time.Ticker must be used instead.
This could result in blocked goroutines as in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .
 2020-02-13 13:21:48 +02:00
Aliaksandr Valialkin
e7d1037210 docs: migrate ExtendedPromQL->MetricsQL in order to be more consistent 2020-02-10 23:03:31 +02:00
Aliaksandr Valialkin
fcdd95a6ef lib/envflag: check for incorrect flag values read from environment vars 2020-02-10 16:09:03 +02:00
Aliaksandr Valialkin
9c5db9400c lib/envflag: add -envflag.enable command-line flag for enabling reading flags from environment vars 
By default flags are read only from command line. They can be read from environment vars if `-envflag.enable` is set.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
 2020-02-10 16:09:01 +02:00
Aliaksandr Valialkin
1010a57882 all: allow setting flags via environment vars 
Now flags can be set via environment vars with the same names as flags.
Command-line flags override flags set via env vars.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
 2020-02-10 13:31:21 +02:00
Aliaksandr Valialkin
ea66212c93 lib/storage: move -dedup.minScrapeInterval flag outside lib/storage, so it doesnt show up in vminsert in cluster version 2020-02-10 13:07:25 +02:00
Aliaksandr Valialkin
8b360a25e9 lib/logger: initialize output to os.Stderr by default 2020-02-04 22:43:26 +02:00
Aliaksandr Valialkin
1f271a9815 lib/logger: add -loggerOutput command-line flag 
This flag allows changing log output from `stderr` to `stdout` if `-loggerOutput=stdout` is set.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/306
 2020-02-04 21:48:24 +02:00
Aliaksandr Valialkin
49ab3fa076 lib/logger: do not clutter -loggerFormat=json output with stack trace 
This should improve json parsing
 2020-02-04 21:40:20 +02:00
Aliaksandr Valialkin
56d6b8ed0a lib/storage: do not deduplicate blocks with less than 32 samples during merge 
This should improve deduplication accuracy for blocks with higher number of samples.
 2020-02-04 18:41:37 +02:00
Aliaksandr Valialkin
7cde594696 all: do not clash flag description with back-quoted flag types 
See https://golang.org/pkg/flag/#PrintDefaults for more details.
 2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin
9b25a2fb67 lib/fs: remove unused readerAt interface 2020-01-31 15:13:00 +02:00
Aliaksandr Valialkin
e3adc095bd all: add -dedup.minScrapeInterval command-line flag for data de-duplication 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
 2020-01-31 01:18:54 +02:00
Aliaksandr Valialkin
a45f25699c lib/storage: re-use indexSearch inside Storage.prefetchMetricNames 2020-01-31 01:18:53 +02:00
Aliaksandr Valialkin
cb5c39ee70 lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor 
This should improve performance when reading many small blocks.
 2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
da19fffa08 all: rename ReadAt* to MustReadAt* in order to dont clash with io.ReaderAt 2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
1332ddc15e lib/storage: pass missing AccountID and ProjectID to searchMetricName 2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
4ed5e9a7ce lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init 
This should speed up Search.NextMetricBlock loop for big number of found time series.
 2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
cb2a2f281f lib/mergeset: properly update lastAccesstime in indexBlockCache entries 
This is a follow-up for 6665f10e7b
 2020-01-29 21:21:01 +02:00
Aliaksandr Valialkin
170c1c3a4e app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value 2020-01-29 00:48:14 +02:00
Aliaksandr Valialkin
a9c1d5b351 app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps 2020-01-28 22:53:50 +02:00
Aliaksandr Valialkin
81ba371eaf lib/logger: fix improperly set skipframes for all the logging functions 2020-01-26 18:34:58 +02:00
Aliaksandr Valialkin
9f595cb2b1 lib/httpserver: log the caller of httpserver.Errorf 
Previously log message contained `httpserver.Errorf`, not it contains the caller of `httpserver.Errorf`, which is more useful.
 2020-01-25 20:18:06 +02:00
Aliaksandr Valialkin
36a1a21d6e lib/protoparser: add parser for Prometheus exposition text format 
This parser will be used by vmagent
 2020-01-24 20:11:19 +02:00
Aliaksandr Valialkin
0cda6afa8e app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent 2020-01-24 16:55:18 +02:00
Aliaksandr Valialkin
ea53a21b02 all: consistently log durations in seconds with millisecond precision 
This should improve logs readability
 2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin
40e564eb9c app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d 2020-01-21 19:05:29 +02:00
Aliaksandr Valialkin
9eaa2ab871 app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp 2020-01-21 15:00:35 +02:00
Aliaksandr Valialkin
62b041e90a lib/{mergeset,storage}: properly update lastAccessTime in index and data block cache entries 2020-01-20 15:00:10 +02:00
Aliaksandr Valialkin
607d4418b8 lib/uint64set: add missing bucket32.b16his values 2020-01-18 14:26:23 +02:00
Aliaksandr Valialkin
e3379537cd lib/uint64set: optimize Set.Union 
This should improve performance for queries over big number of time series
 2020-01-18 13:47:34 +02:00
Aliaksandr Valialkin
5077efd3f7 lib/uint64set: add benchmarks for Set.Union 2020-01-18 13:47:33 +02:00
Aliaksandr Valialkin
a851c75703 lib/storage: skip recovering timestamps order for lossless compression (PrecisionBits=64) 2020-01-17 23:59:19 +02:00
Aliaksandr Valialkin
2084921e64 all: use github.com/klauspost/compress/gzip instead of compress/gzip 
`github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`.
This gives better gzip compression and decompression speeds.
 2020-01-17 23:59:17 +02:00
Aliaksandr Valialkin
ab4d5d72eb lib/uint64set: reduce memory allocations in Set.AppendTo 2020-01-17 22:33:00 +02:00
Aliaksandr Valialkin
476c7fb109 lib/storage: reduce memory allocations when merging metricID sets 2020-01-17 22:10:56 +02:00
Aliaksandr Valialkin
29d21259f0 lib/uint64set: typo fix in Set.Intersect 2020-01-17 18:11:46 +02:00
Aliaksandr Valialkin
ed1d259b10 lib/uint64set: optimize Intersect, Subtract and Union functions 
This should improve performance for queries over big number of time series.
 2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin
68d35357b1 lib/uint64set: improve benchmark for Set.Intersect 2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin
ffe352ad31 lib/uint64set: add benchmark for Set.Intersect 2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin
4b16b7fd11 all: mention command-line flags used for limiting the incoming request size in error messages 
This should improve error logs usability.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287
 2020-01-16 13:06:43 +02:00
Aliaksandr Valialkin
7d429e2806 lib/uint64set: reduce memory usage in Union, Intersect and Subtract methods 
Iterate items with newly added Set.ForEach method instead of allocating `[]uint64`
slice for all the items before the iteration.
 2020-01-15 12:15:48 +02:00
Aliaksandr Valialkin
caffb0cd01 lib/{mergeset,storage}: fix uint64 counters alignment for 32-bit architectures (GOARCH=386, GOARCH=arm) 2020-01-14 22:47:42 +02:00
Aliaksandr Valialkin
b03ccbf6f7 lib/{storage,mergeset}: gradually remove stale entries from block cache and index caches 
This should reduce memory usage in the long run when old blocks and indexes
aren't accessed anymore.
 2020-01-14 21:38:29 +02:00
Aliaksandr Valialkin
bcd3f0c5bd app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions 
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
 2020-01-11 14:47:13 +02:00