2. [Histograms and summaries](https://prometheus.io/docs/practices/histograms/)
3. [How does a Prometheus Histogram work?](https://www.robustperception.io/how-does-a-prometheus-histogram-work)
4. [Improving histogram usability for Prometheus and Grafana](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350)
#### Summary
Summary is quite similar to [histogram](#histogram) and is used for
The visualisation of summaries is pretty straightforward:
{% include img.html href="keyConcepts_summary.png" %}
Such an approach makes summaries easier to use but also puts significant limitations - summaries can't be aggregated.
The [histogram](#histogram) exposes the raw values via counters. It means a user can aggregate these counters
for different metrics (for example, for metrics with different `instance` label) and **then calculate quantiles**.
For summary, quantiles are already calculated, so they [can't be aggregated](https://latencytipoftheday.blogspot.de/2014/06/latencytipoftheday-you-cant-average.html)
with other metrics.
Summaries are usually used for measuring latency, sizes of elements (batch size, for example) etc.
But taking into account the limitation mentioned above.
#### Instrumenting application with metrics
As was said at the beginning of the section [Types of metrics](#types-of-metrics), metric type defines
how it was measured. VictoriaMetrics TSDB doesn't know about metric types, all it sees are labels,
values, and timestamps. And what are these metrics, what do they measure, and how - all this depends
on the application which emits them.
To instrument your application with metrics compatible with VictoriaMetrics TSDB we recommend
using [VictoriaMetrics/metrics](https://github.com/VictoriaMetrics/metrics) instrumentation library.
See more about how to use it on example of
[How to monitor Go applications with VictoriaMetrics](https://victoriametrics.medium.com/how-to-monitor-go-applications-with-victoriametrics-c04703110870)
article.
VictoriaMetrics is also compatible with
Prometheus [client libraries for metrics instrumentation](https://prometheus.io/docs/instrumenting/clientlibs/).
## Write data
There are two main models in monitoring for data collection: [push](#push-model) and [pull](#pull-model).
Both are used in modern monitoring and both are supported by VictoriaMetrics.
### Push model
Push model is a traditional model of the client sending data to the server:
{% include img.html href="keyConcepts_push_model.png" %}
The client (application) decides when and where to send/ingest its metrics.
VictoriaMetrics supports following protocols for ingesting:
* [InfluxDB line protocol](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) over HTTP, TCP and UDP.
* [Graphite plaintext protocol](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-graphite-compatible-agents-such-as-statsd) with [tags](https://graphite.readthedocs.io/en/latest/tags.html#carbon).
* [OpenTSDB put message](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#sending-data-via-telnet-put-protocol).
All the protocols are fully compatible with VictoriaMetrics [data model](#data-model) and can be used in production.
There are no officially supported clients by VictoriaMetrics team for data ingestion.
We recommend choosing from already existing clients compatible with the listed above protocols
(like [Telegraf](https://github.com/influxdata/telegraf) for [InfluxDB line protocol](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)).
Creating custom clients or instrumenting the application for metrics writing is as easy as sending a POST request:
```bash
curl -d '{"metric":{"__name__":"foo","job":"node_exporter"},"values":[0,1,2],"timestamps":[1549891472010,1549891487724,1549891503438]}' -X POST 'http://localhost:8428/api/v1/import'
```
It is allowed to push/write metrics to [Single-server-VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html),
and [vmagent](https://docs.victoriametrics.com/vmagent.html).
The pros of push model:
* application decides how and when to send data;
* with a batch size of which size, at which rate;
* with which retry logic;
* simpler security management, the only access needed for the application is the access to the TSDB.
See [Foiled by the Firewall: A Tale of Transition From Prometheus to VictoriaMetrics](https://www.percona.com/blog/2020/12/01/foiled-by-the-firewall-a-tale-of-transition-from-prometheus-to-victoriametrics/)
elaborating more on why Percona switched from pull to push model.
The cons of push protocol:
* it requires applications to be more complex,
since they need to be responsible for metrics delivery;
* applications need to be aware of monitoring systems;
* using a monitoring system it is hard to tell whether the application
went down or just stopped sending metrics for a different reason;
* applications can overload the monitoring system by pushing
too many metrics.
### Pull model
Pull model is an approach popularized by [Prometheus](https://prometheus.io/),
where the monitoring system decides when and where to pull metrics from:
{% include img.html href="keyConcepts_pull_model.png" %}
In pull model, the monitoring system needs to be aware of all the applications it needs
to monitor. The metrics are scraped (pulled) with fixed intervals via HTTP protocol.
For metrics scraping VictoriaMetrics supports [Prometheus exposition format](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
and needs to be configured with `-promscrape.config` flag pointing to the file with scrape configuration.
This configuration may include list of static `targets` (applications or services)
or `targets` discovered via various service discoveries.
and [vmagent](https://docs.victoriametrics.com/vmagent.html).
The pros of the pull model:
* monitoring system decides how and when to scrape data,
so it can't be overloaded;
* applications aren't aware of the monitoring system and don't need
to implement the logic for delivering metrics;
* the list of all monitored targets belongs to the monitoring system
and can be quickly checked;
* easy to detect faulty or crashed services when they don't respond.
The cons of the pull model:
* monitoring system needs access to applications it monitors;
* the frequency at which metrics are collected depends on the monitoring system.
### Common approaches for data collection
VictoriaMetrics supports both [Push](#push-model) and [Pull](#pull-model)
models for data collection. Many installations are using
exclusively one or second model, or both at once.
The most common approach for data collection is using both models:
{% include img.html href="keyConcepts_data_collection.png" %}
In this approach the additional component is used - [vmagent](https://docs.victoriametrics.com/vmagent.html).
Vmagent is a lightweight agent whose main purpose is to collect and deliver metrics.
It supports all the same mentioned protocols and approaches mentioned for both data collection models.
The basic setup for using VictoriaMetrics and vmagent for monitoring is described
in example of [docker-compose manifest](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker).
In this example, vmagent [scrapes a list of targets](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/prometheus.yml)
and [forwards collected data to VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/9d7da130b5a873be334b38c8d8dec702c9e8fac5/deployment/docker/docker-compose.yml#L15).
VictoriaMetrics is then used as a [datasource for Grafana](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/provisioning/datasources/datasource.yml)
installation for querying collected data.
VictoriaMetrics components allow building more advanced topologies.
For example, vmagents pushing metrics from separate datacenters to the central VictoriaMetrics:
{% include img.html href="keyConcepts_two_dcs.png" %}
VictoriaMetrics in example may be [Single-server-VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html)
or [VictoriaMetrics Cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
Vmagent also allows to fan-out the same data to multiple destinations.
## Query data
VictoriaMetrics provides an [HTTP API](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#prometheus-querying-api-usage)
for serving read queries. The API is used in various integrations such as