The application which accepts the alert rules, executes them on given source, sends(fires) an alert to(in) alert management system
### Components
#### Alert Config Reader
It accepts yaml config as input parameter in Prometheus format, parses it into Go struct.
#### Source Caller
Create own watchdog for every alert group (goroutines), which executes alert query on given source and issues an alert if source returns non-empty result.
Source can be any service which supports PromQL (MetricsQL).
#### Alert Management System Provider
Send positive alert to alert management system, provides interface for every concrete implementation.
Should be ingratiated with Prometheus alertmanager.
open questions:
- do we really need alert group or can just run every alert in own goroutine?
#### Web Server
Expose metrics
open questions:
- should the tool provide API or UI for managing alerting rules? Where to store config updated via the API or UI?
- should the tool provide “alerting rules validation mode” for validating and debugging alerting rules? This mode is useful when creating and debugging alerting rules.
#### Requirements:
- Stateless
- Avoid external dependencies if possible
- Reuse existing code from VictoriaMetrics repo
- Makefile rules for common tasks – see Makefiles for other apps in the app/ dir
- Every package should be covered by tests
- Dockerfile
- Graceful shutdown
- Helm template
- Application uses command line flags for configuration
