Design of a Metrics Monitoring and Alerting System
The design of a metrics monitoring and alerting system involves a structured architecture that transforms raw metric samples into actionable insights through dashboards and alerts. Key components include collectors, a pub-sub system, a time-series database, and an alert manager, all working in tandem to ensure efficient data processing and problem detection. The system supports both push and pull data collection methods, and incorporates AI-assisted features for incident analysis and natural language querying. This design emphasizes scalability, fault tolerance, and the ability to handle high volumes of data while maintaining alert quality and operational efficiency.
In this lesson, we’ll design the architecture that turns raw metric samples into dashboards and timely alerts. We’ll see how collectors, a pub-sub system, a time-series database, a query service, an alert evaluator, an alert manager, and an AI assistant service work together to detect problems quickly and reach the right responders. We’ll begin with the high-level design, expand it into a detailed design, and then add AI-assisted features on top.
High-level design
The following illustration shows the high-level architecture of the system, including both push and pull collection and the separate paths for alerting and historical queries:
At the center of the design is a
Component | Role in the System |
Load balancer | Distributes pushed metric traffic across collectors |
Collectors | Scrape pull targets or receive pushed samples, then validate and normalize them |
Pub-sub system | Buffers samples between collectors and storage and allows replay after failures |
Time-series database | Stores recent samples and rollups for fast writes and range queries |
Blob storage | Holds older, compacted data at a lower cost |
Query service | Runs queries over stored metrics for dashboards and APIs |
Metadata store | Stores alert rules, dashboard definitions, and notification policies |
Alerting service | Checks alert rules against recent data and sends notifications when a rule fires |
Workflow
The following steps show how a sample moves through the system, from collection to alert:
Metric sources either expose metrics for collectors to scrape or push them in batches through the load balancer.
Collectors validate and normalize the samples, then publish them to the pub-sub system.
Storage consumers read the samples from the pub-sub system and write them to the time-series database. As data ages, it is compacted and moved to blob storage.
When an engineer opens a dashboard, the query service reads recent data from the time-series database and older data from blob storage, then returns the results.
The alerting service periodically evaluates rules from the metadata store against recent data in the time-series database. When a rule fires, it sends a notification to the right channel.
Decoupling these stages lets each one scale on its own load, and the pub-sub system absorbs traffic bursts before they reach storage.
API design
Metric sources and users interact with the system through the following APIs. Time-series storage, data retention, and alert evaluation are handled internally, so they don’t need user-facing APIs.
Metrics collection APIs
Write metrics: Sources that use push collection send samples in batches through this API. The writeMetrics() API returns a success or failure status for the batch.
writeMetrics(api_key, samples)
Each item in samples contains a metric name, labels, a timestamp, and a value. Pull collection skips this API, since collectors fetch metrics from an endpoint, usually /metrics, that each source exposes.
Query APIs
Query metrics: Dashboards and users retrieve metric data over a time range through this API. The queryMetrics() API returns the matching time series, with one value per step interval.
queryMetrics(user_id, query, start_time, end_time, step)
For example, the query rate(http_requests_total{status=~"5.."}[5m]) returns the per-second rate of requests with a 5xx status, averaged over a 5-minute window. The =~ operator applies a regular expression, so "5.." matches any three-digit status code that starts with 5.
Alert rule management APIs
Create alert rule: Users define a new alert rule through this API. The createAlertRule() API returns the ID of the new rule.
createAlertRule(user_id, rule_name, expression, duration, severity, channel_id)
Update alert rule: Users change an existing rule’s condition, duration, severity, or notification target. The updateAlertRule() API returns a success or failure status.
updateAlertRule(user_id, rule_id, rule_name, expression, duration, severity, channel_id)
...