docs: add docs for scrape_configs section

app/vmagent/README.md

@@ -259,7 +259,7 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
   scrape_samples_scraped > 10000
   ```
 
-* `scrape_samples_limit` - this metric exposes the configured limit on the number of metrics the given target can expose. The limit can be set via `sample_limit` option at [scrape_configs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config). This allows detecting targets, which expose too many metrics compared to the configured `sample_limit`. For example, the following query returns targets (identified by `instance` label), which expose more than 80% metrics compared to the configed `sample_limit`:
+* `scrape_samples_limit` - this metric exposes the configured limit on the number of metrics the given target can expose. The limit can be set via `sample_limit` option at [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs). This allows detecting targets, which expose too many metrics compared to the configured `sample_limit`. For example, the following query returns targets (identified by `instance` label), which expose more than 80% metrics compared to the configed `sample_limit`:
 
   ```metricsql
   scrape_samples_scraped / scrape_samples_limit > 0.8
@@ -510,8 +510,8 @@ scrape_configs:
 
 Proxy can be configured with the following optional settings:
 
-* `proxy_authorization` for generic token authorization. See [Prometheus docs for details on authorization section](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)
+* `proxy_authorization` for generic token authorization. See [these docs](https://docs.victoriametrics.com/sd_configs.html#http-api-client-options).
-* `proxy_basic_auth` for Basic authorization. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).
+* `proxy_basic_auth` for Basic authorization. See [these docs](https://docs.victoriametrics.com/sd_configs.html#http-api-client-options).
 * `proxy_bearer_token` and `proxy_bearer_token_file` for Bearer token authorization
 * `proxy_oauth2` for OAuth2 config. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#oauth2).
 * `proxy_tls_config` for TLS config. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config).
@@ -546,7 +546,7 @@ By default `vmagent` doesn't limit the number of time series each scrape target
 
 All the scraped metrics are dropped for time series exceeding the given limit. The exceeded limit can be [monitored](#monitoring) via `promscrape_series_limit_rows_dropped_total` metric.
 
-See also `sample_limit` option at [scrape_config section](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).
+See also `sample_limit` option at [scrape_config section](https://docs.victoriametrics.com/sd_configs.html#scrape_configs).
 
 By default `vmagent` doesn't limit the number of time series written to remote storage systems specified at `-remoteWrite.url`. The limit can be enforced by setting the following command-line flags:
 

docs/CHANGELOG.md

@@ -62,7 +62,7 @@ Released at 14-07-2022
 * FEATURE: add `-search.setLookbackToStep` command-line flag, which enables InfluxDB-like gap filling during querying. See [these docs](https://docs.victoriametrics.com/guides/migrate-from-influx.html) for details.
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add an UI for [query tracing](https://docs.victoriametrics.com/#query-tracing). It can be enabled by clicking `trace query` checkbox and re-running the query. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2703).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `-remoteWrite.headers` command-line option for specifying optional HTTP headers to send to the configured `-remoteWrite.url`. For example, `-remoteWrite.headers='Foo:Bar^^Baz:x'` would send `Foo: Bar` and `Baz: x` HTTP headers with every request to `-remoteWrite.url`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805).
-* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): push per-target `scrape_samples_limit` metric to the cofigured `-remoteWrite.url` if `sample_limit` option is set for this target in [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config). See [this feature request](https://github.com/VictoriaMetrics/operator/issues/497).
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): push per-target `scrape_samples_limit` metric to the configured `-remoteWrite.url` if `sample_limit` option is set for this target in [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs). See [this feature request](https://github.com/VictoriaMetrics/operator/issues/497).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): attach node-level labels to [kubernetes_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) targets if `attach_metadata: {"node": true}` is set for `role: endpoints` and `role: endpointslice`. This is a feature backport from Prometheus 2.37 - see [this pull request](https://github.com/prometheus/prometheus/pull/10759).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add ability to specify additional HTTP headers to send to scrape targets via `headers` section in `scrape_configs`. This can be used when the scrape target requires custom authorization and authentication like in [this stackoverflow question](https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header). For example, the following config instructs sending `My-Auth: top-secret` and `TenantID: FooBar` headers with each request to `http://host123:8080/metrics`:
 
@@ -212,7 +212,7 @@ Released at 05-05-2022
 
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for sending data to remote storage with AWS sigv4 authorization. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): allow filtering targets by target url and by target labels with [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) on `http://vmagent:8429/targets` page. This may be useful when `vmagent` scrapes big number of targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1796).
-* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): reduce `-promscrape.config` reload duration when the config contains big number of jobs (aka [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) sections) and only a few of them are changed. Previously all the jobs were restarted. Now only the jobs with changed configs are restarted. This should reduce the probability of data miss because of slow config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270).
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): reduce `-promscrape.config` reload duration when the config contains big number of jobs (aka [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) sections) and only a few of them are changed. Previously all the jobs were restarted. Now only the jobs with changed configs are restarted. This should reduce the probability of data miss because of slow config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): improve service discovery speed for big number of scrape targets. This should help when `vmagent` discovers big number of targets (e.g. thousands) in Kubernetes cluster. The service discovery speed now should scale with the number of CPU cores available to `vmagent`.
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add ability to attach node-level labels and annotations to discovered Kubernetes pod targets in the same way as Prometheus 2.35 does. See [this feature request](https://github.com/prometheus/prometheus/issues/9510) and [this pull request](https://github.com/prometheus/prometheus/pull/10080).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for `tls_config` and `proxy_url` options at `oauth2` section in the same way as Prometheus does. See [oauth2 docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#oauth2).
@@ -564,7 +564,7 @@ Released at 22-10-2021
 * BUGFIX: vmalert, vmauth: prevent from frequent closing of TCP connections established to backends under high load. This should reduce the number of TCP sockets in `TIME_WAIT` state at `vmalert` and `vmauth` under high load. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1704).
 * BUGFIX: vmalert: correctly calculate alert ID including extra labels. Previously, ID for alert entity was generated without alertname or groupname. This led to collision, when multiple alerting rules within the same group producing same labelsets. E.g. expr: `sum(metric1) by (job) > 0` and expr: `sum(metric2) by (job) > 0` could result into same labelset `job: "job"`. The bugfix adds all extra labels right after receiving response from the datasource. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1734).
 * BUGFIX: vmalert: fix links in [Web UI](https://docs.victoriametrics.com/vmalert.html#web). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1717).
-* BUGFIX: vmagent: set `honor_timestamps: true` by default in [scrape configs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) if this options isn't set explicitly. This aligns the behaviour with Prometheus.
+* BUGFIX: vmagent: set `honor_timestamps: true` by default in [scrape configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) if this options isn't set explicitly. This aligns the behaviour with Prometheus.
 * BUGFIX: vmagent: group scrape targets by the original job names at `http://vmagent:8429/targets` page like Prometheus does. Previously they were grouped by the job name after relabeling, which may result in unexpected empty target groups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1707).
 * BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl.html): fix importing boolean fields from InfluxDB line protocol. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1709).
 

docs/FAQ.md

@@ -39,7 +39,7 @@ Yes in most cases. VictoriaMetrics can substitute Prometheus in the following as
 ## What is the difference between vmagent and Prometheus?
 
 While both [vmagent](https://docs.victoriametrics.com/vmagent.html) and Prometheus may scrape Prometheus targets (aka `/metrics` pages)
-according to the provided Prometheus-compatible [scrape configs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)
+according to the provided Prometheus-compatible [scrape configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs)
 and send data to multiple remote storage systems, vmagent has the following additional features:
 
 * vmagent usually requires lower amounts of CPU, RAM and disk IO compared to Prometheus when scraping an enormous number of targets (more than 1000)

docs/operator/additional-scrape.MD

@@ -8,8 +8,7 @@ AdditionalScrapeConfigs is an additional way to add scrape targets in VMAgent CR
 There are two options for adding targets into VMAgent: inline configuration into CRD or defining it as a Kubernetes Secret.
 
 No validation happens during the creation of configuration. However, you must validate job specs, and it must follow job spec configuration.
-Please check official Prometheus documentation as references.
+Please check [scrape_configs documentation](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) as references.
-[Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).
 
 ## Inline Additional Scrape Configuration in VMAgent CRD
 

docs/sd_configs.md

@@ -977,7 +977,7 @@ scrape_configs:
     # The `http://host:port/metrics` endpoint is scraped per each configured traget then.
     # The `http` scheme can be changed to `https` by setting it via `scheme` field at `scrape_config` level.
     # The `/metrics` path can be changed to arbitrary path via `metrics_path` field at `scrape_config` level.
-    # See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config .
+    # See https://docs.victoriametrics.com/sd_configs.html#scrape_configs .
     #
     # Alternatively the scheme and path can be changed via `relabel_configs` section at `scrape_config` level.
     # See https://docs.victoriametrics.com/vmagent.html#relabeling .
@@ -1128,3 +1128,148 @@ The following additional options can be specified in the majority of supported s
     # proxy_tls_config:
     #   ...
 ```
+
+
+## scrape_configs
+
+The `scrape_configs` section at file pointed by `-promscrape.config` command-line flag can contain [supported service discovery options](#service-discovery).
+Additionally, it can contain the following options:
+
+```yaml
+scrape_configs:
+  # job_name must contain value for `job` label, which is added
+  # to all the metrics collected from the configured and discovered scrape targets.
+  # See https://prometheus.io/docs/concepts/jobs_instances/ .
+- job_name: "..."
+
+  # scrape_interval is an optional interval to scrape targets.
+  # By default the scrape_interval sepcified in `global` section is used.
+  # See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file
+  # If `global` section doesn't contain the `scrape_interval` option,
+  # then one minute interval is used.
+  # Example values:
+  # - "30s" - 30 seconds
+  # - "2m" - 2 minutes
+  # scrape_interval: <duration>
+
+  # scrape_timeout is an optional timeout when scraping the targets.
+  # By default the scrape_timeout specified in `global` section is used.
+  # See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file
+  # If `global` section doesn't contain the `scrape_timeout` option,
+  # then 10 seconds interval is used.
+  # Example values:
+  # - "30s" - 30 seconds
+  # - "2m" - 2 minutes
+  # The `scrape_timeout` cannot exceed the `scrape_interval`.
+  # scrape_timeout: <duration>
+
+  # metrics_path is the path to fetch metrics from targets.
+  # By default metrics are fetched from "/metrics" path.
+  # metrics_path: "..."
+
+  # honor_labels controls how to handle conflicts between labels that are
+  # already present in scraped data and labels that would be attached
+  # server-side ("job" and "instance" labels, manually configured target
+  # labels, and labels generated by service discovery implementations).
+  #
+  # If honor_labels is set to "true", label conflicts are resolved by keeping label
+  # values from the scraped data and ignoring the conflicting server-side labels.
+  #
+  # If honor_labels is set to "false", label conflicts are resolved by renaming
+  # conflicting labels in the scraped data to "exported_<original-label>" (for
+  # example "exported_instance", "exported_job") and then attaching server-side
+  # labels.
+  #
+  # Setting honor_labels to "true" is useful for use cases such as federation and
+  # scraping the Pushgateway, where all labels specified in the target should be
+  # preserved.
+  #
+  # By default honor_labels is set to false for security and consistency reasons.
+  # honor_labels: <boolean>
+
+  # honor_timestamps controls whether to respect the timestamps present in scraped data.
+  #
+  # If honor_timestamps is set to "true", the timestamps of the metrics exposed
+  # by the target will be used.
+  #
+  # If honor_timestamps is set to "false", the timestamps of the metrics exposed
+  # by the target will be ignored.
+  #
+  # By default honor_timestamps is set to true.
+  # honor_timestamps: <boolean>
+
+  # scheme configures the protocol scheme used for requests.
+  # Supported values: http and https.
+  # By default http is used.
+  # scheme: "..."
+
+  # Optional query arg parameters to add to scrape url.
+  # params:
+  #   "param_name1": ["value1", ..., "valueN"]
+  #   ...
+  #   "param_nameM": ["valueM1", ..., "valueMN"]
+
+  # relabel_configs is an optional relabeling configurations
+  # for the specified and discovered scrape targets.
+  # See https://docs.victoriametrics.com/vmagent.html#relabeling
+  # relabel_configs:
+  # - <relabel_config> ...
+
+  # metric_relabel_configs is an optional relabeling configs
+  # for the collected metrics from active scrape targets.
+  # See https://docs.victoriametrics.com/vmagent.html#relabeling
+  # metric_relabel_configs:
+  # - <relabel_config> ...
+
+  # sample_limit is an optional per-scrape limit on number
+  # of scraped samples that will be accepted.
+  # If more than this number of samples are present after metric relabeling
+  # the entire scrape will be treated as failed.
+  # By default the limit is disabled.
+  # sample_limit: <int>
+
+  # relabel_debug enables debugging for relabel_configs if set to true.
+  # See https://docs.victoriametrics.com/vmagent.html#relabeling
+  # relabel_debug: <boolean>
+
+  # metric_relabel_debug enables debugging for metric_relabel_configs if set to true.
+  # See https://docs.victoriametrics.com/vmagent.html#relabeling
+  # metric_relabel_debug: <boolean>
+
+  # disable_compression allows disabling HTTP compression for responses received from scrape targets.
+  # By default scrape targets are queried with `Accept-Encoding: gzip` http request header,
+  # so targets could send compressed responses in order to save network bandwidth.
+  # See https://docs.victoriametrics.com/vmagent.html#scrape_config-enhancements
+  # disable_compression: <boolean>
+
+  # disable_keepalive allows disabling HTTP keep-alive when scraping targets.
+  # By default HTTP keep-alive is enabled, so TCP connections to scrape targets
+  # could be re-used.
+  # See https://docs.victoriametrics.com/vmagent.html#scrape_config-enhancements
+  # disable_keepalive: <boolean>
+
+  # stream_parse allows enabling stream parsing mode when scraping targets.
+  # By default stram parsing mode is disabled for targets which return up to a few thosands samples.
+  # See https://docs.victoriametrics.com/vmagent.html#stream-parsing-mode .
+  # stream_parse: <boolean>
+
+  # scrape_align_interval allows aligning scrapes to the given interval.
+  # Example values:
+  # - "5m" - align scrapes to every 5 minutes.
+  # - "1h" - align scrapes to every hour.
+  # See https://docs.victoriametrics.com/vmagent.html#scrape_config-enhancements
+  # scrape_align_interval: <duration>
+
+  # scrape_offset allows specifying the exact offset for scrapes.
+  # Example values:
+  # - "5m" - align scrapes to every 5 minutes.
+  # - "1h" - align scrapes to every hour.
+  # See https://docs.victoriametrics.com/vmagent.html#scrape_config-enhancements
+  # scrape_offset: <duration>
+
+  # series_limit is an optional limit on the number of unique time series
+  # a single target can expose during all the scrapes.
+  # By default there is no limit on the number of exposed series.
+  # See https://docs.victoriametrics.com/vmagent.html#cardinality-limiter .
+  # series_limit: ...
+```

docs/vmagent.md

@@ -263,7 +263,7 @@ Extra labels can be added to metrics collected by `vmagent` via the following me
   scrape_samples_scraped > 10000
   ```
 
-* `scrape_samples_limit` - this metric exposes the configured limit on the number of metrics the given target can expose. The limit can be set via `sample_limit` option at [scrape_configs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config). This allows detecting targets, which expose too many metrics compared to the configured `sample_limit`. For example, the following query returns targets (identified by `instance` label), which expose more than 80% metrics compared to the configed `sample_limit`:
+* `scrape_samples_limit` - this metric exposes the configured limit on the number of metrics the given target can expose. The limit can be set via `sample_limit` option at [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs). This allows detecting targets, which expose too many metrics compared to the configured `sample_limit`. For example, the following query returns targets (identified by `instance` label), which expose more than 80% metrics compared to the configed `sample_limit`:
 
   ```metricsql
   scrape_samples_scraped / scrape_samples_limit > 0.8
@@ -514,8 +514,8 @@ scrape_configs:
 
 Proxy can be configured with the following optional settings:
 
-* `proxy_authorization` for generic token authorization. See [Prometheus docs for details on authorization section](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)
+* `proxy_authorization` for generic token authorization. See [these docs](https://docs.victoriametrics.com/sd_configs.html#http-api-client-options).
-* `proxy_basic_auth` for Basic authorization. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).
+* `proxy_basic_auth` for Basic authorization. See [these docs](https://docs.victoriametrics.com/sd_configs.html#http-api-client-options).
 * `proxy_bearer_token` and `proxy_bearer_token_file` for Bearer token authorization
 * `proxy_oauth2` for OAuth2 config. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#oauth2).
 * `proxy_tls_config` for TLS config. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config).
@@ -550,7 +550,7 @@ By default `vmagent` doesn't limit the number of time series each scrape target
 
 All the scraped metrics are dropped for time series exceeding the given limit. The exceeded limit can be [monitored](#monitoring) via `promscrape_series_limit_rows_dropped_total` metric.
 
-See also `sample_limit` option at [scrape_config section](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).
+See also `sample_limit` option at [scrape_config section](https://docs.victoriametrics.com/sd_configs.html#scrape_configs).
 
 By default `vmagent` doesn't limit the number of time series written to remote storage systems specified at `-remoteWrite.url`. The limit can be enforced by setting the following command-line flags: