Problem Statement: Metrics Are the Nervous System of Production
Frames the monitoring stack as a write-heavy time-series platform with query, visualization, and paging responsibilities, not a generic dashboard app.
Problem statement
Design a centralized monitoring and alerting platform that scrapes metrics from tens of thousands of service instances, stores them in a time-series database (Prometheus TSDB), lets engineers build and query dashboards (Grafana), and pages on-call humans through Alertmanager when thresholds or anomalies fire. The platform must serve multiple engineering teams with self-service dashboards and rules, survive regional failures, and scale to millions of samples per second.
Metrics are numeric time series: a metric name plus a set of key-value labels identifies one series, and each scrape appends one sample (timestamp, value) per series. This shape drives everything: append-only writes at enormous rate, reads that are narrow range scans over recent data, and a unique killer - label cardinality explosion.
Roblox reported roughly 100 billion samples per day across 100-plus Prometheus servers at PromCon 2018, and Google's Monarch paper (EuroSys 2020) describes collecting tens of billions of samples per second globally. These are published reference points; every uncited number in this answer is an explicit design assumption.
The four functional requirements from the brief: a centralized TSDB, service discovery for scrape targets, Grafana dashboards and queries, and alert rules with threshold or anomaly detection. The four non-functional requirements: millions of data points per second, HA or clustering, retention and disk management, and federation across Prometheus servers.
Key Highlights
- •A sample is (timestamp, value) appended to one series identified by metric name plus label set.
- •Writes are append-only and dominate; reads are range queries dominated by the last hours.
- •Roblox publicly reported ~100B samples/day on 100+ Prometheus servers; Google Monarch reports tens of billions of samples/sec.
- •Label cardinality explosion, not raw throughput, is the most common production killer.
- •The stack has four planes: collection, storage, query/visualization, and alerting/notification.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate collection, storage, query, and alerting because they scale and fail independently."
- "Before choosing any component, let me pin down series count, sample rate, and retention."