Design Monitoring & Alerting - System Design | WinJob

Problem Statement: Metrics, Alerts, and SLOs at Scale

Why this matters for monitoring and alerting

Interviewers at Datadog, New Relic, and Prometheus-native shops expect you to connect collection, storage, query, and notification with explicit SLO language—not a box diagram of "metrics go to Grafana."

Core mechanisms (Prometheus + Grafana + Alertmanager)

Pull scrape with service discovery keeps apps ignorant of collector topology; push paths (remote write, OTLP) exist for short-lived jobs.
TSDB block storage with compaction and retention tiers bounds disk; federation or Thanos/Mimir scales query beyond one Prometheus.
Alert pipeline separates rule evaluation (Prometheus) from routing, inhibition, and paging (Alertmanager) so on-call noise is policy, not code.

Operational invariants

Label cardinality is a budget — every new label dimension multiplies series count; treat user_id on HTTP metrics as a design bug unless sampled.
Alerts fire on SLO burn, not raw CPU spikes — tie pages to customer-visible failure modes and multi-window burn rates.
Observability is multi-tenant safe — RBAC on metrics, dashboards, and silences must align with service ownership.

Interview signal

Quantify samples/sec, active series, retention days, and query p95. State RPO for metrics (acceptable gap during AZ failure) and alert delivery latency (<60s for sev-1). Mention runbooks and auto-silence during deploys.

Phase focus (`understanding`)

Section 1 deepens understanding decisions.

javaOne Dark Pro

1 // Illustrative: burn-rate alert gate (conceptual)
2 public final class SloBurnGate {
3   private final double errorBudgetFraction;
4   public boolean pageOnBurn(double burnRate5m, double burnRate1h) {
5     return burnRate5m > 14 * errorBudgetFraction
6         || burnRate1h > 6 * errorBudgetFraction;
7   }
8 }

pythonOne Dark Pro

1 def page_on_burn(burn_5m: float, burn_1h: float, budget: float) -> bool:
2     return burn_5m > 14 * budget or burn_1h > 6 * budget

typescriptOne Dark Pro

1 export function pageOnBurn(burn5m: number, burn1h: number, budget: number): boolean {
2   return burn5m > 14 * budget || burn1h > 6 * budget;
3 }

How to open this one

The framing that lands for monitoring is connecting collection, storage, query, and notification with explicit SLO language rather than drawing a box for Grafana. Lead with pull-based scrape plus service discovery (apps stay ignorant of collector topology), a TSDB with retention tiers, and alerting on symptoms not causes — burn-rate alerts over raw thresholds. The failure story that matters: a metrics-pipeline outage blinds the alerting path, so you alert on the absence of data too. That shows you treat observability as a system with its own failure modes.

Key Highlights

•Cardinality is a first-class capacity dimension
•Separate metric evaluation from alert routing
•Page on SLO burn, not single-threshold CPU
•Federation/HA paths must preserve query correctness

1	// Illustrative: burn-rate alert gate (conceptual)
2	public final class SloBurnGate {
3	private final double errorBudgetFraction;
4	public boolean pageOnBurn(double burnRate5m, double burnRate1h) {
5	return burnRate5m > 14 * errorBudgetFraction
6	\|\| burnRate1h > 6 * errorBudgetFraction;
7	}
8	}

1	def page_on_burn(burn_5m: float, burn_1h: float, budget: float) -> bool:
2	return burn_5m > 14 * budget or burn_1h > 6 * budget

1	export function pageOnBurn(burn5m: number, burn1h: number, budget: number): boolean {
2	return burn5m > 14 * budget \|\| burn1h > 6 * budget;
3	}