Design a Monitoring & Alerting Stack (Prometheus + Grafana)

Hard45 min
1 / 30
System design answer

Design a Monitoring & Alerting Stack (Prometheus + Grafana)

Hard45 min30 sections
understanding•7 min read

Problem Statement: Metrics Are the Nervous System of Production

Frames the monitoring stack as a write-heavy time-series platform with query, visualization, and paging responsibilities, not a generic dashboard app.

Problem statement

Design a centralized monitoring and alerting platform that scrapes metrics from tens of thousands of service instances, stores them in a time-series database (Prometheus TSDB), lets engineers build and query dashboards (Grafana), and pages on-call humans through Alertmanager when thresholds or anomalies fire. The platform must serve multiple engineering teams with self-service dashboards and rules, survive regional failures, and scale to millions of samples per second.

Metrics are numeric time series: a metric name plus a set of key-value labels identifies one series, and each scrape appends one sample (timestamp, value) per series. This shape drives everything: append-only writes at enormous rate, reads that are narrow range scans over recent data, and a unique killer - label cardinality explosion.

Roblox reported roughly 100 billion samples per day across 100-plus Prometheus servers at PromCon 2018, and Google's Monarch paper (EuroSys 2020) describes collecting tens of billions of samples per second globally. These are published reference points; every uncited number in this answer is an explicit design assumption.

The four functional requirements from the brief: a centralized TSDB, service discovery for scrape targets, Grafana dashboards and queries, and alert rules with threshold or anomaly detection. The four non-functional requirements: millions of data points per second, HA or clustering, retention and disk management, and federation across Prometheus servers.

Key Highlights

  • •A sample is (timestamp, value) appended to one series identified by metric name plus label set.
  • •Writes are append-only and dominate; reads are range queries dominated by the last hours.
  • •Roblox publicly reported ~100B samples/day on 100+ Prometheus servers; Google Monarch reports tens of billions of samples/sec.
  • •Label cardinality explosion, not raw throughput, is the most common production killer.
  • •The stack has four planes: collection, storage, query/visualization, and alerting/notification.
Lead With Cardinality
Say in the first two minutes that unbounded label values (user IDs, request IDs) can multiply series count and kill any TSDB. This instantly marks you as someone who has run Prometheus in production.
Do Not Draw One Box
A single Prometheus instance handling millions of samples per second with 2-year retention does not exist. The design must show sharding, replication, and long-term storage from the start.

Section Rescue Kit

Buzzwords to use:

Time Series IdentitySample

Safe statements:

  • "I will separate collection, storage, query, and alerting because they scale and fail independently."
  • "Before choosing any component, let me pin down series count, sample rate, and retention."
Design a Monitoring & Alerting Stack (Prometheus + Grafana) - System Design | WinJob | WinJob