Add sysctl docs to README.md

Signed-off-by: Johannes Ziemke <github@5pi.de>

README.md

@@ -124,7 +124,6 @@ selinux | Exposes SELinux statistics. | Linux
 sockstat | Exposes various statistics from `/proc/net/sockstat`. | Linux
 softnet | Exposes statistics from `/proc/net/softnet_stat`. | Linux
 stat | Exposes various statistics from `/proc/stat`. This includes boot time, forks and interrupts. | Linux
-sysctl | Expose sysctl values from `/proc/sys`. Use `--collector.sysctl.include(-info)` to configure. | Linux
 tapestats | Exposes statistics from `/sys/class/scsi_tape`. | Linux
 textfile | Exposes statistics read from local disk. The `--collector.textfile.directory` flag must be set. | _any_
 thermal | Exposes thermal statistics like `pmset -g therm`. | Darwin
@@ -156,6 +155,35 @@ and does not time out.  In addition, monitor the
 `scrape_samples_post_metric_relabeling` metric to see the changes in
 cardinality.
 
+Name     | Description | OS
+---------|-------------|----
+buddyinfo | Exposes statistics of memory fragments as reported by /proc/buddyinfo. | Linux
+cgroups | A summary of the number of active and enabled cgroups | Linux
+devstat | Exposes device statistics | Dragonfly, FreeBSD
+drbd | Exposes Distributed Replicated Block Device statistics (to version 8.4) | Linux
+ethtool | Exposes network interface information and network driver statistics equivalent to `ethtool`, `ethtool -S`, and `ethtool -i`. | Linux
+interrupts | Exposes detailed interrupts statistics. | Linux, OpenBSD
+ksmd | Exposes kernel and system statistics from `/sys/kernel/mm/ksm`. | Linux
+lnstat | Exposes stats from `/proc/net/stat/`. | Linux
+logind | Exposes session counts from [logind](http://www.freedesktop.org/wiki/Software/systemd/logind/). | Linux
+meminfo\_numa | Exposes memory statistics from `/proc/meminfo_numa`. | Linux
+mountstats | Exposes filesystem statistics from `/proc/self/mountstats`. Exposes detailed NFS client statistics. | Linux
+network_route | Exposes the routing table as metrics | Linux
+ntp | Exposes local NTP daemon health to check [time](./docs/TIME.md) | _any_
+perf | Exposes perf based metrics (Warning: Metrics are dependent on kernel configuration and settings). | Linux
+processes | Exposes aggregate process statistics from `/proc`. | Linux
+qdisc | Exposes [queuing discipline](https://en.wikipedia.org/wiki/Network_scheduler#Linux_kernel) statistics | Linux
+runit | Exposes service status from [runit](http://smarden.org/runit/). | _any_
+slabinfo | Exposes slab statistics from `/proc/slabinfo`. Note that permission of `/proc/slabinfo` is usually 0400, so set it appropriately. | Linux
+supervisord | Exposes service status from [supervisord](http://supervisord.org/). | _any_
+sysctl | Expose sysctl values from `/proc/sys`. Use `--collector.sysctl.include(-info)` to configure. | Linux
+systemd | Exposes service and system status from [systemd](http://www.freedesktop.org/wiki/Software/systemd/). | Linux
+tcpstat | Exposes TCP connection status information from `/proc/net/tcp` and `/proc/net/tcp6`. (Warning: the current version has potential performance issues in high load situations.) | Linux
+wifi | Exposes WiFi device and station statistics. | Linux
+zoneinfo | Exposes NUMA memory zone metrics. | Linux
+
+### Perf Collector
+
 The `perf` collector may not work out of the box on some Linux systems due to kernel
 configuration and security settings. To allow access, set the following `sysctl`
 parameter:
@@ -191,33 +219,56 @@ found using [`perf list`](http://man7.org/linux/man-pages/man1/perf.1.html) or
 from debugfs. And example usage of this would be
 `--collector.perf.tracepoint="sched:sched_process_exec"`.
 
+### Sysctl Collector
 
-Name     | Description | OS
+The `sysctl` collector can be enabled with `--collector.sysctl`. It supports exposing numeric sysctl values
----------|-------------|----
+as metrics using the `--collector.sysctl.include` flag and string values as info metrics by using the
-buddyinfo | Exposes statistics of memory fragments as reported by /proc/buddyinfo. | Linux
+`--collector.sysctl.include-info` flag. The flags can be repeated. For sysctl with multiple numeric values,
-cgroups | A summary of the number of active and enabled cgroups | Linux
+an optional mapping can be given to expose each value as its own metric. Otherwise an `index` label is used
-devstat | Exposes device statistics | Dragonfly, FreeBSD
+to identify the different fields.
-drbd | Exposes Distributed Replicated Block Device statistics (to version 8.4) | Linux
-ethtool | Exposes network interface information and network driver statistics equivalent to `ethtool`, `ethtool -S`, and `ethtool -i`. | Linux
-interrupts | Exposes detailed interrupts statistics. | Linux, OpenBSD
-ksmd | Exposes kernel and system statistics from `/sys/kernel/mm/ksm`. | Linux
-lnstat | Exposes stats from `/proc/net/stat/`. | Linux
-logind | Exposes session counts from [logind](http://www.freedesktop.org/wiki/Software/systemd/logind/). | Linux
-meminfo\_numa | Exposes memory statistics from `/proc/meminfo_numa`. | Linux
-mountstats | Exposes filesystem statistics from `/proc/self/mountstats`. Exposes detailed NFS client statistics. | Linux
-network_route | Exposes the routing table as metrics | Linux
-ntp | Exposes local NTP daemon health to check [time](./docs/TIME.md) | _any_
-perf | Exposes perf based metrics (Warning: Metrics are dependent on kernel configuration and settings). | Linux
-processes | Exposes aggregate process statistics from `/proc`. | Linux
-qdisc | Exposes [queuing discipline](https://en.wikipedia.org/wiki/Network_scheduler#Linux_kernel) statistics | Linux
-runit | Exposes service status from [runit](http://smarden.org/runit/). | _any_
-slabinfo | Exposes slab statistics from `/proc/slabinfo`. Note that permission of `/proc/slabinfo` is usually 0400, so set it appropriately. | Linux
-supervisord | Exposes service status from [supervisord](http://supervisord.org/). | _any_
-systemd | Exposes service and system status from [systemd](http://www.freedesktop.org/wiki/Software/systemd/). | Linux
-tcpstat | Exposes TCP connection status information from `/proc/net/tcp` and `/proc/net/tcp6`. (Warning: the current version has potential performance issues in high load situations.) | Linux
-wifi | Exposes WiFi device and station statistics. | Linux
-zoneinfo | Exposes NUMA memory zone metrics. | Linux
 
+#### Examples
+##### Numeric values
+###### Single values
+Using `--collector.sysctl.include=vm.user_reserve_kbytes`:
+`vm.user_reserve_kbytes = 131072` -> `node_sysctl_vm_user_reserve_kbytes 131072`
+
+###### Multiple values
+A sysctl can contain multiple values, for example:
+```
+net.ipv4.tcp_rmem = 4096	131072	6291456
+```
+Using `--collector.sysctl.include=net.ipv4.tcp_rmem` the collector will expose:
+```
+node_sysctl_net_ipv4_tcp_rmem{index="0"} 4096
+node_sysctl_net_ipv4_tcp_rmem{index="1"} 131072
+node_sysctl_net_ipv4_tcp_rmem{index="2"} 6291456
+```
+If the indexes have defined meaning like in this case, the values can be mapped to multiple metrics by appending the mapping to the --collector.sysctl.include flag:
+Using `--collector.sysctl.include=net.ipv4.tcp_rmem:min,default,max` the collector will expose:
+```
+node_sysctl_net_ipv4_tcp_rmem_min 4096
+node_sysctl_net_ipv4_tcp_rmem_default 131072
+node_sysctl_net_ipv4_tcp_rmem_max 6291456
+```
+
+##### String values
+String values need to be exposed as info metric. The user selects them by using the `--collector.sysctl.include-info` flag.
+
+###### Single values
+`kernel.core_pattern = core` -> `node_sysctl_info{key="kernel.core_pattern_info", value="core"} 1`
+
+###### Multiple values
+Given the following sysctl:
+```
+kernel.seccomp.actions_avail = kill_process kill_thread trap errno trace log allow
+```
+Setting `--collector.sysctl.include-info=kernel.seccomp.actions_avail` will yield:
+```
+node_sysctl_info{key="kernel.seccomp.actions_avail", index="0", value="kill_process"} 1
+node_sysctl_info{key="kernel.seccomp.actions_avail", index="1", value="kill_thread"} 1
+...
+```
 
 ### Textfile Collector