update wiki pages

CHANGELOG.md

@@ -20,6 +20,8 @@ The following tip changes can be tested by building VictoriaMetrics components f
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `--kafka.consumer.topic.concurrency` command-line flag. It controls the number of Kafka consumer workers to use by `vmagent`. It should eliminate the need to start multiple `vmagent` instances to improve data transfer rate. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for [Kafka producer and consumer](https://docs.victoriametrics.com/vmagent.html#kafka-integration) on `arm64` machines. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271).
 
+* BUGFIX: prevent from slow [snapshot creating](https://docs.victoriametrics.com/#how-to-work-with-snapshots) under high data ingestion rate. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551).
+
 ## [v1.89.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.89.1)
 
 Released at 2023-03-12

guides/grafana-vmgateway-openid-configuration.md

@@ -34,38 +34,41 @@ See details about all supported options in the [vmgateway documentation](https:/
 [Keycloak](https://www.keycloak.org/) is an open source identity service that can be used to issue JWT tokens.
 
 1. Log in with admin credentials to your Keycloak instance
-2. Go to `Clients` -> `Create`.
+2. Go to `Clients` -> `Create`.<br>
-   Use `OpenID Connect` as `Client Type`.
+   Use `OpenID Connect` as `Client Type`.<br>
-   Specify `grafana` as `Client ID`.
+   Specify `grafana` as `Client ID`.<br>
-   Click `Next`.
+   Click `Next`.<br>
    <img src="grafana-vmgateway-openid-configuration/create-client-1.png" width="800">
-3. Enable `Client authentication`.
+3. Enable `Client authentication`.<br>
-   Enable `Authorization`.
+   Enable `Authorization`.<br>
-   <img src="grafana-vmgateway-openid-configuration/create-client-2.png" width="800">
+   <img src="grafana-vmgateway-openid-configuration/create-client-2.png" width="800"><br>
-   Click `Next`.
+   Click `Next`.<br>
-4. Add Grafana URL as `Valid Redirect URIs`. For example, `http://localhost:3000/`.
+4. Add Grafana URL as `Root URL`. For example, `http://localhost:3000/`.<br>
-   <img src="grafana-vmgateway-openid-configuration/create-client-3.png" width="800">
+   <img src="grafana-vmgateway-openid-configuration/create-client-3.png" width="800"><br>
-   Click `Save`.
+   Click `Save`.<br>
-5. Go to `Clients` -> `grafana` -> `Credentials`.
+5. Go to `Clients` -> `grafana` -> `Credentials`.<br>
-   <img src="grafana-vmgateway-openid-configuration/client-secret.png" width="800">
+   <img src="grafana-vmgateway-openid-configuration/client-secret.png" width="800"><br>
-   Copy the value of `Client secret`. It will be used later in Grafana configuration.
+   Copy the value of `Client secret`. It will be used later in Grafana configuration.<br>
-6. Go to `Clients` -> `grafana` -> `Client scopes`.
+6. Go to `Clients` -> `grafana` -> `Client scopes`.<br>
-   Click at `grafana-dedicated` -> `Add mapper`.
+   Click at `grafana-dedicated` -> `Add mapper` -> `By configuration` -> `User attribute`.<br>
-   <img src="grafana-vmgateway-openid-configuration/create-mapper-1.png" width="800">
+   <img src="grafana-vmgateway-openid-configuration/create-mapper-1.png" width="800"><br>
-   <img src="grafana-vmgateway-openid-configuration/create-mapper-2.png" width="800">
+   <img src="grafana-vmgateway-openid-configuration/create-mapper-2.png" width="800"><br>
-   Configure the mapper as follows
+   Configure the mapper as follows<br>
-    - `Mapper Type` as `User Attribute`.
     - `Name` as `vm_access`.
     - `Token Claim Name` as `vm_access`.
     - `User Attribute` as `vm_access`.
     - `Claim JSON Type` as `JSON`.
-      Enable `Add to ID token` and `Add to access token`.
+      Enable `Add to ID token` and `Add to access token`.<br>
-      <img src="grafana-vmgateway-openid-configuration/create-mapper-3.png" width="800">
+   
-      Click `Save`.
+    <img src="grafana-vmgateway-openid-configuration/create-mapper-3.png" width="800"><br>
-7. Go to `Users` -> select user to configure claims -> `Attributes`.
+    Click `Save`.<br>
-   Specify `vm_access` as `Key`.
+7. Go to `Users` -> select user to configure claims -> `Attributes`.<br>
-   Specify `{"tenant_id" : {"account_id": 0, "project_id": 0 }}` as `Value`.
+   Specify `vm_access` as `Key`.<br>
-   <img src="grafana-vmgateway-openid-configuration/user-attributes.png" width="800">
+   For the purpose of this example, we will use 2 users:<br>
+   - for the first user we will specify `{"tenant_id" : {"account_id": 0, "project_id": 0 },"extra_labels":{ "team": "admin" }}` as `Value`.
+   - for the second user we will specify `{"tenant_id" : {"account_id": 0, "project_id": 1 },"extra_labels":{ "team": "dev" }}` as `Value`.
+   <br>
+   <img src="grafana-vmgateway-openid-configuration/user-attributes.png" width="800"><br>
    Click `Save`.
 
 ## Configure grafana
@@ -187,8 +190,146 @@ URL should point to the vmgateway instance.
 You can also use VictoriaMetrics [Grafana datasource](https://github.com/VictoriaMetrics/grafana-datasource) plugin.
 See installation instructions [here](https://github.com/VictoriaMetrics/grafana-datasource#installation).
 
-Enable `Forward OAuth identity` flag.
+Enable `Forward OAuth identity` flag.<br>
 <img src="grafana-vmgateway-openid-configuration/grafana-ds.png" width="800">
 
 Now you can use Grafana to query metrics from the specified tenant.
 Users with `vm_access` claim will be able to query metrics from the specified tenant.
+
+## Test multi-tenant access
+
+For the test purpose we will setup the following services as [docker-compose](https://docs.docker.com/compose/) manifest:
+- Grafana
+- Keycloak
+- vmagent to generate test metrics
+- VictoriaMetrics cluster
+- vmgateway configured to work in cluster mode
+- VictoriaMetrics single node
+- vmgateway configured to work in single node mode
+
+```yaml
+version: '3'
+
+services:
+  keycloak:
+    image: quay.io/keycloak/keycloak:21.0
+    command:
+      - start-dev
+    ports:
+      - 3001:8080
+    environment:
+      KEYCLOAK_ADMIN: admin
+      KEYCLOAK_ADMIN_PASSWORD: change_me
+
+  grafana:
+    image: grafana/grafana-oss:9.4.3
+    network_mode: host
+    volumes:
+      - ./grafana.ini:/etc/grafana/grafana.ini
+      - grafana_data:/var/lib/grafana/
+
+  vmsingle:
+    image: victoriametrics/victoria-metrics:v1.89.1
+    command:
+      - -httpListenAddr=0.0.0.0:8429
+
+  vmstorage:
+    image: victoriametrics/vmstorage:v1.89.1-cluster
+
+  vminsert:
+    image: victoriametrics/vminsert:v1.89.1-cluster
+    command:
+      - -storageNode=vmstorage:8400
+      - -httpListenAddr=0.0.0.0:8480
+
+  vmselect:
+    image: victoriametrics/vmselect:v1.89.1-cluster
+    command:
+      - -storageNode=vmstorage:8401
+      - -httpListenAddr=0.0.0.0:8481
+
+  vmagent:
+    image: victoriametrics/vmagent:v1.89.1
+    volumes:
+      - ./scrape.yaml:/etc/vmagent/config.yaml
+    command:
+      - -promscrape.config=/etc/vmagent/config.yaml
+      - -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write
+      - -remoteWrite.url=http://vmsingle:8429/api/v1/write
+
+  vmgateway-cluster:
+    image: victoriametrics/vmgateway:v1.89.1-enterprise
+    ports:
+      - 8431:8431
+    command:
+      - -eula
+      - -enable.auth=true
+      - -clusterMode=true
+      - -write.url=http://vminsert:8480
+      - -read.url=http://vmselect:8481
+      - -httpListenAddr=0.0.0.0:8431
+      - -auth.oidcDiscoveryEndpoints=http://keycloak:8080/realms/master/.well-known/openid-configuration
+
+  vmgateway-single:
+    image: victoriametrics/vmgateway:v1.89.1-enterprise
+    ports:
+      - 8432:8431
+    command:
+      - -eula
+      - -enable.auth=true
+      - -write.url=http://vmsingle:8429
+      - -read.url=http://vmsingle:8429
+      - -httpListenAddr=0.0.0.0:8431
+      - -auth.oidcDiscoveryEndpoints=http://keycloak:8080/realms/master/.well-known/openid-configuration
+
+volumes:
+  grafana_data:
+```
+
+For the test purpose vmagent will be configured to scrape metrics from the following targets(`scrape.yaml` contents):
+
+```yaml
+scrape_configs:
+  - job_name: stat
+    metric_relabel_configs:
+      - if: "{instance =~ 'vmgateway.*'}"
+        action: replace
+        target_label: team
+        replacement: admin
+      - if: "{instance =~ 'localhost.*'}"
+        action: replace
+        target_label: team
+        replacement: dev
+    static_configs:
+      - targets:
+          - localhost:8429
+          - vmgateway-single:8431
+          - vmgateway-cluster:8431
+```
+
+Relabeling rules will add the `team` label to the scraped metrics in order to test multi-tenant access.
+Metrics from `localhost` will be labeled with `team=dev` and metrics from `vmgateway` will be labeled with `team=admin`.
+
+vmagent will write data into VictoriaMetrics single-node and cluster(with tenant `0:0`).
+
+Grafana datasources configuration will be the following:
+
+<img src="grafana-vmgateway-openid-configuration/grafana-test-datasources.png" width="800">
+
+Let's login as user with `team=dev` labels limitation set via claims.
+
+Using `vmgateway-cluster` results into `No data` response as proxied request will go to tenant `0:1`.
+Since vmagent is only configured to write to `0:0` `No data` is an expected response.
+
+<img src="grafana-vmgateway-openid-configuration/dev-cluster-nodata.png" width="800">
+
+Switching to `vmgateway-single` does have data. Note that it is limited to metrics with `team=dev` label.
+
+<img src="grafana-vmgateway-openid-configuration/dev-single-data.png" width="800">
+
+Now lets login as user with `team=admin`.
+
+Both cluster and single node datasources now return metrics for `team=admin`.
+
+<img src="grafana-vmgateway-openid-configuration/admin-cluster-data.png" width="800">
+<img src="grafana-vmgateway-openid-configuration/admin-single-data.png" width="800">

operator/api.md

@@ -65,6 +65,8 @@ This Document documents the types introduced by the VictoriaMetrics to be consum
 * [HTTPAuth](#httpauth)
 * [ServiceSpec](#servicespec)
 * [StorageSpec](#storagespec)
+* [StreamAggrConfig](#streamaggrconfig)
+* [StreamAggrRule](#streamaggrrule)
 * [VMAlert](#vmalert)
 * [VMAlertDatasourceSpec](#vmalertdatasourcespec)
 * [VMAlertList](#vmalertlist)
@@ -709,6 +711,7 @@ VMAgentRemoteWriteSpec defines the remote storage configuration for VmAgent
 | tlsConfig | TLSConfig describes tls configuration for remote write target | *[TLSConfig](#tlsconfig) | false |
 | sendTimeout | Timeout for sending a single block of data to -remoteWrite.url (default 1m0s) | *string | false |
 | headers | Headers allow configuring custom http headers Must be in form of semicolon separated header with value e.g. headerName: headerValue vmagent supports since 1.79.0 version | []string | false |
+| streamAggrConfig | StreamAggrConfig defines stream aggregation configuration for VMAgent for -remoteWrite.url | *[StreamAggrConfig](#streamaggrconfig) | false |
 
 [Back to TOC](#table-of-contents)
 
@@ -744,6 +747,7 @@ VMAgentSpec defines the desired state of VMAgent
 | dnsPolicy | DNSPolicy set DNS policy for the pod | [v1.DNSPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#pod-v1-core) | false |
 | topologySpreadConstraints | TopologySpreadConstraints embedded kubernetes pod configuration option, controls how pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ | [][v1.TopologySpreadConstraint](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) | false |
 | scrapeInterval | ScrapeInterval defines how often scrape targets by default | string | false |
+| scrapeTimeout | ScrapeTimeout defines global timeout for targets scrape | string | false |
 | aPIServerConfig | APIServerConfig allows specifying a host and auth methods to access apiserver. If left empty, VMAgent is assumed to run inside of the cluster and will discover API servers automatically and use the pod's CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/. | *[APIServerConfig](#apiserverconfig) | false |
 | overrideHonorLabels | OverrideHonorLabels if set to true overrides all user configured honor_labels. If HonorLabels is set in ServiceScrape or PodScrape to true, this overrides honor_labels to false. | bool | false |
 | overrideHonorTimestamps | OverrideHonorTimestamps allows to globally enforce honoring timestamps in all scrape configs. | bool | false |
@@ -958,6 +962,34 @@ StorageSpec defines the configured storage for a group Prometheus servers. If ne
 
 [Back to TOC](#table-of-contents)
 
+## StreamAggrConfig
+
+StreamAggrConfig defines the stream aggregation config
+
+| Field | Description | Scheme | Required |
+| ----- | ----------- | ------ | -------- |
+| rules | Stream aggregation rules | [][StreamAggrRule](#streamaggrrule) | true |
+| keepInput | Allows writing both raw and aggregate data | bool | false |
+| dedupInterval | Allows setting different de-duplication intervals per each configured remote storage | string | false |
+
+[Back to TOC](#table-of-contents)
+
+## StreamAggrRule
+
+StreamAggrRule defines the rule in stream aggregation config
+
+| Field | Description | Scheme | Required |
+| ----- | ----------- | ------ | -------- |
+| match | Match is a label selector for filtering time series for the given selector.\n\nIf the match isn't set, then all the input time series are processed. | string | false |
+| interval | Interval is the interval between aggregations. | string | true |
+| outputs | Outputs is a list of output aggregate functions to produce.\n\nThe following names are allowed:\n\n- total - aggregates input counters - increase - counts the increase over input counters - count_series - counts the input series - count_samples - counts the input samples - sum_samples - sums the input samples - last - the last biggest sample value - min - the minimum sample value - max - the maximum sample value - avg - the average value across all the samples - stddev - standard deviation across all the samples - stdvar - standard variance across all the samples - histogram_bucket - creates VictoriaMetrics histogram for input samples - quantiles(phi1, ..., phiN) - quantiles' estimation for phi in the range [0..1]\n\nThe output time series will have the following names:\n\n  input_name:aggr_<interval>_<output> | []string | true |
+| by | By is an optional list of labels for grouping input series.\n\nSee also Without.\n\nIf neither By nor Without are set, then the Outputs are calculated individually per each input time series. | []string | false |
+| without | Without is an optional list of labels, which must be excluded when grouping input series.\n\nSee also By.\n\nIf neither By nor Without are set, then the Outputs are calculated individually per each input time series. | []string | false |
+| input_relabel_configs | InputRelabelConfigs is an optional relabeling rules, which are applied on the input before aggregation. | [][RelabelConfig](#relabelconfig) | false |
+| output_relabel_configs | OutputRelabelConfigs is an optional relabeling rules, which are applied on the aggregated output before being sent to remote storage. | [][RelabelConfig](#relabelconfig) | false |
+
+[Back to TOC](#table-of-contents)
+
 ## VMAlert
 
 VMAlert  executes a list of given alerting or recording rules against configured address.
@@ -1198,6 +1230,7 @@ VMSingleSpec defines the desired state of VMSingle
 | nodeSelector | NodeSelector Define which Nodes the Pods are scheduled on. | map[string]string | false |
 | terminationGracePeriodSeconds | TerminationGracePeriodSeconds period for container graceful termination | *int64 | false |
 | readinessGates | ReadinessGates defines pod readiness gates | []v1.PodReadinessGate | false |
+| streamAggrConfig | StreamAggrConfig defines stream aggregation configuration for VMSingle | *[StreamAggrConfig](#streamaggrconfig) | false |
 
 [Back to TOC](#table-of-contents)