Design Incident Management - System Design | WinJob

Problem Statement: Incidents, On-Call, and Escalation at Scale

Why this matters for incident management

Interviewers at PagerDuty, Opsgenie, and Splunk On-Call (VictorOps) expect you to connect ingestion, correlation, escalation, and audit with explicit on-call SLOs—not a box labeled "send Slack message."

Core mechanisms

Normalized events from Prometheus, CloudWatch, Datadog, and custom webhooks land on an idempotent ingestion tier with signature verification.
Correlation merges alerts into incidents using fingerprint keys, time windows, and optional ML clustering so on-call sees one actionable object.
Escalation policies separate who is on-call (schedules) from how we notify (parallel/serial steps, channel mix, timeouts).

Operational invariants

One incident per customer-visible failure mode — duplicate pages are a design defect; use grouping and inhibition across services.
Acknowledgement stops escalation clocks — unacked sev-1 must escalate within minutes with auditable timeline.
Notifications are at-least-once — dedupe keys on provider message ids; voice fallback when SMS fails.

Phase focus (`understanding`)

PagerDuty-class platforms turn noisy signals into owned, time-bounded incidents with auditable response.

Interview signal

State events/min, incidents/day, notification p95, and escalation depth. Mention runbook links on every page, status page linkage, and postmortem export.

javaOne Dark Pro

1 // Illustrative: escalation step gate (conceptual)
2 public final class EscalationGate {
3   private final Duration stepDelay;
4   public boolean shouldEscalate(Instant lastNotify, Instant now, boolean acked) {
5     return !acked && Duration.between(lastNotify, now).compareTo(stepDelay) >= 0;
6   }
7 }

pythonOne Dark Pro

1 def should_escalate(last_notify: float, now: float, delay_s: int, acked: bool) -> bool:
2     return not acked and (now - last_notify) >= delay_s

typescriptOne Dark Pro

1 export function shouldEscalate(
2   lastNotifyMs: number,
3   nowMs: number,
4   delayMs: number,
5   acked: boolean,
6 ): boolean {
7   return !acked && nowMs - lastNotifyMs >= delayMs;
8 }

How to open this one

The framing that lands for incident management is the lifecycle from detect to resolve to learn: alert routing and deduplication, escalation policies with on-call schedules, a single source of truth for incident state, and a blameless postmortem that feeds back into the system. Lead with alert deduplication — a thundering herd of alerts for one root cause must collapse to one incident — and the failure story that proves it: alert fatigue causes a real page to be missed, so severity and routing have to be ruthless. That shows you understand the goal is shrinking time-to-resolve, not collecting alerts.

Key Highlights

•Correlate before paging
•Escalation policies are time contracts
•Notifications are at-least-once with dedupe
•Audit timeline is immutable for postmortems

1	// Illustrative: escalation step gate (conceptual)
2	public final class EscalationGate {
3	private final Duration stepDelay;
4	public boolean shouldEscalate(Instant lastNotify, Instant now, boolean acked) {
5	return !acked && Duration.between(lastNotify, now).compareTo(stepDelay) >= 0;
6	}
7	}

1	def should_escalate(last_notify: float, now: float, delay_s: int, acked: bool) -> bool:
2	return not acked and (now - last_notify) >= delay_s

1	export function shouldEscalate(
2	lastNotifyMs: number,
3	nowMs: number,
4	delayMs: number,
5	acked: boolean,
6	): boolean {
7	return !acked && nowMs - lastNotifyMs >= delayMs;
8	}