Kiu/VictoriaMetrics
Kiu/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-16 00:41:24 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
2,422 Commits 114 Branches 748 Tags 509 MiB
894246176f
Commit Graph

942 Commits

Author SHA1 Message Date
Aliaksandr Valialkin
9e644ef111 lib/httpserver: make sure the gzipResponseWriter.Write() is called on Flush() and Close() calls 
This should fix the `http: superfluous response.WriteHeader call` issue

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1078
 2021-02-28 19:23:26 +02:00
Aliaksandr Valialkin
9a2bf65134 lib/promscrape: add ability to spread scrape targets among multiple vmagent instances 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084
 2021-02-28 18:40:42 +02:00
Aliaksandr Valialkin
3e44d9947e lib/promscrape/discovery/kubernetes: properly account the number of objects when watcher is stopped 
A follow-up for b21b110b7a
 2021-02-28 17:06:49 +02:00
Aliaksandr Valialkin
0ef7a94056 lib/promscrape/discovery/kubernetes: add vm_promscrape_discovery_kubernetes_* metrics for monitoring internal state of k8s service discovery 2021-02-28 16:58:45 +02:00
Aliaksandr Valialkin
f52bdbe2a3 lib/promscrape/discovery/kubernetes: remove resourceVersionMatch=NotOlderThan query arg when watching for k8s object changes, since it cannot be used when watch=1 query arg is passed 2021-02-28 16:08:43 +02:00
Aliaksandr Valialkin
6f8866a40a lib/promscrape: fix possible deadlock in parallel execution of target relabeling 2021-02-28 16:05:32 +02:00
Aliaksandr Valialkin
5c9e657808 lib/promscrape/discovery/kubernetes: fix deadlock in startWatcherForURL 
reloadObjects must be called without holding aw.mu lock
 2021-02-28 15:25:33 +02:00
Aliaksandr Valialkin
e77f2f8630 lib/promscrape/discovery/kubernetes: typo fix after 241ffd1f3b 2021-02-28 15:15:27 +02:00
Aliaksandr Valialkin
82441537ff lib/promscrape/discovery/kubernetes: pre-populate labelsByKey in reloadObject() 2021-02-28 15:09:43 +02:00
Aliaksandr Valialkin
e003453941 lib/promscrape/discovery/kubernetes: compare sorted sets of labels in tests 
This should deflake tests where the order of labels isn't stable
 2021-02-28 14:12:32 +02:00
Aliaksandr Valialkin
6a21ef87b7 lib/promscrape: add missing startWatchersForRole() call at the beginning of apiWatcher.getLabelsForRole 2021-02-28 14:00:00 +02:00
Aliaksandr Valialkin
6d0e7fb8b0 lib/promscrape/discovery/kubernetes: reload k8s resources on every error 
This is needed for obtaining fresh resourceVersion
 2021-02-27 01:46:59 +02:00
Aliaksandr Valialkin
7f1302688f lib/fs: follow-up after f3a03c4164 2021-02-27 01:09:37 +02:00
Nikolay
d88fa5ebe4 Adds windows build (#1040) 
* fixes windows compilation,
adds signal impl for windows,
adds free space usage for windows,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036

NOTE victoria metrics database still CANNOT work under windows system,
only vmagent is supported.
To completly port victoria metrics, you have to fix issues with separators,
parsing and posix file removall

* rollback separator

* Adds windows setInformation api,
it must behave like unix, need to test it.
changes procutil

* check for invlaid param

* Fixes posix delete semantic

* refactored a bit

* fixes openbsd build

* removed windows api call

* Fixes code after windows add

* Update lib/procutil/signal_windows.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-02-27 01:06:22 +02:00
Aliaksandr Valialkin
44975e28fe lib/promscrape: yet another typo fix after ed8441ec52 2021-02-26 23:36:34 +02:00
Aliaksandr Valialkin
c1b8729bd8 lib/fs: properly handle stale NFS file handle error during file deletion 
This error can appear when -storageDataPath points to NFS volume and the given file has been already removed.
 2021-02-26 23:24:46 +02:00
Aliaksandr Valialkin
50e74c439c lib/promscrape: typo fix after ed8441ec52 2021-02-26 23:04:40 +02:00
Aliaksandr Valialkin
fa3ce450fb lib/promscrape: cache ScrapeWork 
This should reduce the time needed for updating big number of scrape targets.
 2021-02-26 21:43:41 +02:00
Aliaksandr Valialkin
efcdf613c2 lib/promscrape/discovery/kubernetes: cache target labels 
This should reduce CPU usage on repeated SDConfig.GetLabels() calls.
 2021-02-26 20:24:29 +02:00
Aliaksandr Valialkin
22822feea3 lib/promscrape/discovery/kubernetes: errcheck fix 2021-02-26 19:09:12 +02:00
Aliaksandr Valialkin
dc8c045378 lib/promscrape: cleanup after 9b2246c29b 
Main points:

* Revert changes outside lib/promscrape/discovery/kuberntes . These changes can be applied later in a separate commit
* Minimize changes in lib/promscrape/discovery/kubernetes compared to a93e644001
* Corner case fixes.
 2021-02-26 19:09:12 +02:00
Nikolay
cf9262b01f vmagent kubernetes watch stream discovery. (#1082) 
* started work on sd for k8s

* continue work on watch sd

* fixes

* continue work

* continue work on sd k8s

* disable gzip

* fixes typos

* log errror

* minor fix

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-02-26 19:09:12 +02:00
Aliaksandr Valialkin
4cfac70cde lib/promscrape: remove duplicate code a bit 2021-02-26 15:54:07 +02:00
Aliaksandr Valialkin
55eb983193 lib/promscrape: reduce processing time for big number of discovered targets by processing them in parallel 2021-02-26 12:48:48 +02:00
Aliaksandr Valialkin
f8baf1a76d lib/promrelabel: optimize labeldrop and labelkeep relabeling for prefix.* and prefix.+ regexps 2021-02-24 17:58:11 +02:00
Aliaksandr Valialkin
06676c8feb lib/storage: consistency renaming: durationsPerDateTagFilterCache -> loopsPerDateTagFilterCache 2021-02-23 15:50:08 +02:00
faceair
b1409a7413 lib/storage: correct tagfilter match cost (#1079) 2021-02-22 21:54:37 +02:00
Aliaksandr Valialkin
197ecca426 lib/promrelabel: add more optimizations for relabeling for common cases 2021-02-22 16:36:54 +02:00
Aliaksandr Valialkin
63c16c3fdf lib/promrelabel: optimize relabeling performance for common cases 2021-02-22 00:51:07 +02:00
Aliaksandr Valialkin
8b87398333 lib/promscrape: export vm_promscrape_target_relabel_duration_seconds metric 2021-02-21 23:21:35 +02:00
Aliaksandr Valialkin
587132555f lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation 
This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now,
so GC doesn't need visiting them.
 2021-02-21 22:09:10 +02:00
Aliaksandr Valialkin
bd3bcdc43c lib/storage: do not re-calculate stats for heavy tag filters 
This should reduce the number of slow queries when stats for heavy tag filters was recalculated.
 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
b8a5ee2e93 lib/{mergeset,storage}: allow merging smaller number of small parts 
While this may increase CPU and disk IO usage needed for background merge,
this also recudes CPU usage during queries in production. This is because
such queries tend to read recently added data and it is better to have lower number
of parts for such data in order to reduce CPU usage.

This partially reverts ebf8da3730
 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
34195218e1 lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
f09b46b2b1 lib/promscrape: typo fix after the commit f26162ec99 2021-02-19 00:34:18 +02:00
Aliaksandr Valialkin
72eef964d9 app/vmagent: properly perform graceful shutdown, which was broken in the commit 1d1ba889fe 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
 2021-02-19 00:34:17 +02:00
Aliaksandr Valialkin
502d0e2524 lib/promscrape: add scrape_align_interval config option into scrape config 
This option allows aligning scrapes to a particular intervals.
 2021-02-18 23:53:04 +02:00
Aliaksandr Valialkin
10ccb92e4d lib/storage: use composite index for a query with a name filter and negative filters 2021-02-18 18:57:45 +02:00
Aliaksandr Valialkin
418de71509 lib/storage: properly handle queries containing a filter on metric name plus any number of negative filters and zero non-negative filters 
Example: `node_cpu_seconds_total{mode!="idle"}`
 2021-02-18 18:33:05 +02:00
Aliaksandr Valialkin
628b8eb55e lib/storage: prevent from running identical heavy tag filters in concurrent queries when measuring the number of loops for such tag filter. 
This should reduce CPU usage spikes when measuring the number of loops needed for heavy tag filters
 2021-02-18 14:01:18 +02:00
Aliaksandr Valialkin
fd41f070db lib/storage: sort tag filters by the number of loops they need for the execution 
This metric should work better than the filter execution duration, since it cannot be distorted
by concurrently running queries.
 2021-02-18 12:52:29 +02:00
Aliaksandr Valialkin
9566015a36 Revert "lib/mergeset: tune lifetime for entries inside block caches" 
This reverts commit 458c89324d.

Production testing revealed zero improvements for memory usage with reduced lifetime for entries in block caches.
 2021-02-17 20:42:15 +02:00
Aliaksandr Valialkin
b4c7d5992b lib/storage: move composite filters to the top during sorting 2021-02-17 20:26:32 +02:00
Aliaksandr Valialkin
2005d8212a lib/storage: return back filter arg to getMetricIDsForTagFilter function 
The filter arg has been removed in the commit c7ee2fabb8
because it was preventing from caching the number of matching time series per each tf.

Now the cache contains duration for tf execution, so the filter shouldn't break such caching.
 2021-02-17 19:33:15 +02:00
Aliaksandr Valialkin
83da939947 app/vmstorage: export vm_composite_filter_success_conversions_total and vm_composite_filter_missing_conversions_total metrics 2021-02-17 19:13:49 +02:00
Aliaksandr Valialkin
0c5bb2a397 lib/storage: revert ecf132933e, since negative filters require the same amount of work as non-negative filters 2021-02-17 18:56:13 +02:00
Aliaksandr Valialkin
bbc287ea6a lib/storage: tag filters sorting... 2021-02-17 18:56:12 +02:00
Aliaksandr Valialkin
f5e841c1e9 lib/storage: further tune tag filters sorting 2021-02-17 17:28:36 +02:00
Aliaksandr Valialkin
9d1f14d94c lib/storage: tune the logic for sorting tag filters according the their exeuction times 2021-02-17 15:02:19 +02:00
Aliaksandr Valialkin
1a19702d92 lib/storage: make sure that nobody uses partitions when closing the table 2021-02-17 15:02:18 +02:00
Aliaksandr Valialkin
3062ff0fdb app/vmselect: export per-tenant stats on the number of requests and the cumulative request duration 
The metrics are:
- vm_vmselect_http_requests_total{accountID="...",projectID="..."} - the total number of select requests per each tenant
- vm_vmselect_http_duration_ms_total{accountID="...",projectID="..."} - the total duration in milliseconds for per-tenant select requests

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932
 2021-02-16 23:30:29 +02:00
Aliaksandr Valialkin
3b8596bfec lib/httpserver: make errcheck happy 2021-02-16 22:05:59 +02:00
Aliaksandr Valialkin
1ab7a8dfd5 lib/storage: more tuning for tag filters sorting according the time they take 2021-02-16 21:27:27 +02:00
Aliaksandr Valialkin
35a23234ca lib/mergeset: tune lifetime for entries inside block caches 
This should reduce memory usage in general case without significant CPU usage increase
 2021-02-16 18:12:32 +02:00
Aliaksandr Valialkin
500acb958c lib/mergeset: clarify comments in the code a bit 2021-02-16 18:03:11 +02:00
Aliaksandr Valialkin
da05c10544 lib/uint64set: remove memory allocation in bucket16.appendTo when sorting smallPool 2021-02-16 15:31:59 +02:00
Aliaksandr Valialkin
18305caadb lib/httpserver: cache /metrics output for a second 
This should reduce CPU load when `/metrics` output is scraped with a frequency exceeding a request per second
 2021-02-16 14:57:13 +02:00
Aliaksandr Valialkin
7869d38043 lib/protoparser/influx: make sure that escaped whitespace can be put in measurement, tag names and field names 2021-02-16 13:59:11 +02:00
Aliaksandr Valialkin
d788876a7a lib/mergeset: remove unused code after a4140de9e6 2021-02-16 13:40:29 +02:00
Aliaksandr Valialkin
55952f8f2e lib/storage: tune sorting for tag filters 2021-02-16 13:07:42 +02:00
Aliaksandr Valialkin
3eae03a337 lib/storage: increase match cost for negative tag filters, since they need to scan all the label pairs 2021-02-15 16:39:52 +02:00
Aliaksandr Valialkin
46e98ed490 vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 
The new version switches from log-linear histograms to log-based histograms,
which provide up to 3.6 times better accuracy.
 2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin
93ff866e91 lib/storage: reduce the minimum supported retention for inverted index from one month to one day 2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin
5d5d310dcc lib/flagutil: prevent from integer overflow when parsing duration 2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin
fccb481de2 lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpoints_label_* and __meta_kuberntes_endpoints_annotation_* labels to role: endpoints 
This syncs kubernetes SD with Prometheus 2.25
See 617c56f55a
 2021-02-15 02:51:36 +02:00
Aliaksandr Valialkin
54a09de037 lib/logger: explicitly import "time/tzdata" package for embedding tzdata into the app 
The approach with `timetzdata` build tag didn't work for GOARCH=arm and GOARCH=ppc64le
due to the issue https://github.com/golang/go/issues/44073#issuecomment-778854298
 2021-02-15 01:00:30 +02:00
Aliaksandr Valialkin
6f3bbf21b8 lib/storage: sort tag filters by actual execution time instead of by the number of matching time series 
This should improve query speed for queries with regexp filters matching small number of time series
on a label with big number of unique values.
 2021-02-15 00:19:46 +02:00
Aliaksandr Valialkin
9e3993c585 lib/storage: properly hanle regexp tag filters with dots, which can be converted to full string match filters. 
For example `{label=~"foo\.bar"}` should be converted to `{label="foo.bar"}`. Previously it has was mistakenly conveted to `{label="foo\.bar"}` .
This could result in missing time series for such tag filters.
 2021-02-14 23:39:19 +02:00
Aliaksandr Valialkin
18c2075159 lib/promscrape: remove vm_promscrape_scrapes_failed_per_url_total and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total metrics 
These metrics may result in big number of time series when vmagent scrapes thousands of targets and these targets constantly changes.

* It is better using `up == 0` query for determining failing targets.
* It is better using the following query for determining targets with exceeded limit on the number of metrics:

  scrape_samples_scraped > 0 if up == 0
 2021-02-12 05:23:27 +02:00
Aliaksandr Valialkin
4e645a5fd3 lib/storage: return back in-order applying of tag filters, since concurrently executing tag filters can result in CPU and RAM waste in common case 2021-02-10 22:43:07 +02:00
Aliaksandr Valialkin
eeb92eb7fc lib/storage: load metadata before loading indexdb, since indexdb depends on the metadata 2021-02-10 17:55:51 +02:00
Aliaksandr Valialkin
08f21d8761 app/vmstorage: export vm_composite_index_min_timestamp metric 2021-02-10 17:14:00 +02:00
Aliaksandr Valialkin
b27288f1b0 lib/storage: parallelize tag filters execution a bit 
This should reduce execution time when a query contains multiple tag filters and each such filter matches big number of time series.
 2021-02-10 16:32:27 +02:00
Aliaksandr Valialkin
4262c2f7c2 lib/storage: remove filter arg from getMetricIDsForDateTagFilter function 
The `filter` arg breaks the logic for sorting tag filters by the matching metrics,
which may result in non-optimal performance during time series search.
 2021-02-10 16:32:26 +02:00
Aliaksandr Valialkin
681dfb7485 lib/storage: fix inconsistencies in error logs 2021-02-10 16:32:21 +02:00
Aliaksandr Valialkin
148422bcba lib/storage: disable composite index usage when querying old data 2021-02-10 14:57:58 +02:00
Aliaksandr Valialkin
17d5a03f6e lib/storage: fix metric name match for composite filter 2021-02-10 01:27:34 +02:00
Aliaksandr Valialkin
fa0ef143b1 lib/storage: optimize search by label filters matching big number of time series 2021-02-10 00:46:17 +02:00
Aliaksandr Valialkin
5c9715a89a lib/storage: reduce lock contention in dateMetricIDCache when registering new time series for the current day 
This should help systems with multiple CPU cores
 2021-02-10 00:04:19 +02:00
Aliaksandr Valialkin
082eabf51e lib/fs: remove the code for tracking whether the given memory region is in page cache 
This code didn't give performance gains under production workload, so let's remove it in order to simplify the code.
 2021-02-09 16:51:11 +02:00
Aliaksandr Valialkin
afa9cf9c57 lib/mergeset: remove dead code left after a4140de9e6 2021-02-09 16:51:09 +02:00
Aliaksandr Valialkin
9ed7789fef optimize Storage.updatePerDateData() 2021-02-09 02:59:53 +02:00
Aliaksandr Valialkin
ea328b7391 lib/storage: skip deduplication when creating inmemory data blocks 
The deduplication will be performed later during merging such blocks.
 2021-02-09 02:26:16 +02:00
Aliaksandr Valialkin
7b7963a77f lib/mergeset: unconditionally cache indexdb blocks 
Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage.
This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.
 2021-02-09 00:49:59 +02:00
Aliaksandr Valialkin
e8ee9fa7fe app/vmstorage: export missing vm_cache_size_bytes metrics for indexdb and data caches 2021-02-09 00:49:58 +02:00
Aliaksandr Valialkin
2b36eb3d82 lib/cgroup: follow-up after b9bf3cbe3e 2021-02-08 16:01:26 +02:00
Nikolay
32603f57cc refactored cgroups limits, (#1061) 
adds tests, remove os.Exec
 2021-02-08 16:01:26 +02:00
Aliaksandr Valialkin
2dbb12563b lib/storage: optimize data ingestion in the beginning of every hour 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
 2021-02-08 12:04:51 +02:00
Aliaksandr Valialkin
1a6eb0c3cf lib/logger: exit the app if unsupported timezone value has been passed to -loggerTimezone 
While at it, clarify descrption for `-loggerTimezone` command-line flag.
 2021-02-07 23:33:35 +02:00
Aliaksandr Valialkin
c6a7288109 lib/storage: check for prevHourMetricIDs cache before falling back to checking for (date, metricID) entries during data ingestion 
This should reduce possible CPU usage spikes at the beginning of every hour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
 2021-02-04 18:46:23 +02:00
Aliaksandr Valialkin
b5eba70595 lib/httpserver: expose process_open_fds and process_max_fds metrics 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/402
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1037
 2021-02-04 16:42:32 +02:00
Nikolay
6fc2a8e544 fixes dockerswarm (#1053) 
fixes improper usage of host network services
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1028
 2021-02-04 15:57:41 +02:00
Aliaksandr Valialkin
8249f13104 app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards 
Example: foo{bar{x,yz},a[b-c],*de}
 2021-02-03 20:16:28 +02:00
Aliaksandr Valialkin
2976ec89b8 lib/storage: fix a bug, which breaks searching by Graphite wildcard filters 2021-02-03 20:15:50 +02:00
Aliaksandr Valialkin
45a63a1da9 sort orSuffixes in tagFilter.InitFromGraphiteQuery for faster seeks 2021-02-03 20:15:37 +02:00
Aliaksandr Valialkin
4b930b9ffe app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax 2021-02-03 01:17:19 +02:00
Aliaksandr Valialkin
755b0998ce lib/promscrape: add vm_promscrape_service_discovery_duration_seconds metric 2021-02-02 16:16:51 +02:00
Aliaksandr Valialkin
4c59dbc127 lib/promscrape: add vm_promscrape_scrape_retries_total, vm_promscrape_discovery_retries_total and vm_promscrape_discovery_requests_total metrics 2021-02-01 20:06:16 +02:00
Aliaksandr Valialkin
e79799a784 lib/flagutil: typo fix in comment to ArrayInt.GetOptionalArgOrDefault() func 2021-02-01 14:42:15 +02:00
Aliaksandr Valialkin
fdf9de98f8 app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values 
This commit also adds --vm-round-digits command-line option to vmctl tool.
 2021-02-01 14:42:15 +02:00