Design Health Check System - System Design | WinJob

Problem Statement: Unified Health Control Plane

Context for section 1 (understanding)

Platforms at Kubernetes, AWS, Consul scale must treat jittered probe schedules and failure threshold hysteresis as explicit contracts—not ad-hoc per-team /healthz paths. This section explains problem statement: unified health control plane when fleets exceed 81,500 registered targets with 6s default intervals.

Mechanisms you should articulate

Registry stores probe definitions versioned per service; dry-run evaluates policy before adapters sync.
Data path Consul → Agent → TTL → Catalog → Mesh enforces timeout 600ms, failure threshold 3, success threshold 2.
Control plane emits immutable routing events; adapters never guess—UNKNOWN fails closed for production traffic.

Operational invariants

Readiness gates traffic — dependency checks belong here, not on liveness restart paths.
Evidence before routing — missing probe data yields UNKNOWN, not implicit HEALTHY.
Adapter precedence is documented — when ELB, Consul, and Kubernetes disagree, ops has a single winner table.

Deep dive (understanding focus)

Interviewers probe how Problem Statement: Unified Health Control Plane avoids jittered probe schedules: cap executor fan-out, isolate tenants, and measure SLIs (probe success %, flap rate, adapter sync lag p99).

Quantify 6,120 executions/sec, Redis hot-set memory, and 2448 status read QPS from load balancers.

Mention audit: who changed probe policy, which artifact version, which routing flip fired.

Architecture signals

Draw Consul → Agent → TTL → Catalog → Mesh with failure arrows: executor timeout → retry with jitter; aggregator quorum loss → UNKNOWN; adapter backlog → throttle sync.

Contrast shallow liveness (process up) vs deep readiness (dependencies + feature flags) in separate sentences—never merge them.

Edge cases

Thundering herd after deploy — jitter intervals per target shard.
IPv6-only targets behind legacy LB — probe path must match family.
Large response bodies on /healthz — cap bytes read to prevent OOM.

Metrics snapshot

Metric	Target	Notes
Probe success	99.9%	Excludes deliberate maintenance windows
Flap rate	< 0.5% / day	After hysteresis tuning
Adapter sync lag	p99 < 2s	ELB + EndpointSlice
UNKNOWN rate	< 0.1%	Fail-closed incidents only

pythonOne Dark Pro

1 from dataclasses import dataclass
2  
3 @dataclass
4 class ProbePolicy:
5     interval_s: int
6     timeout_ms: int
7     failure_threshold: int
8     success_threshold: int
9  
10 def should_remove_from_lb(state: str) -> bool:
11     return state in {"UNHEALTHY", "UNKNOWN"}

How to open this one

The framing that signals depth on a health-check system is distinguishing liveness from readiness: liveness says restart me if I am wedged, readiness says route traffic to me only when I can serve — conflating them causes outages. Lead with why a failing readiness probe must pull a pod from rotation without killing it, and the failure story that proves it: a dependency blip flips liveness instead of readiness and a restart storm takes down healthy pods. That shows you understand health checks govern both traffic and restarts, and getting the semantics wrong amplifies failures.

Key Highlights

•Readiness gates traffic; liveness gates restart
•UNKNOWN fail-closed for missing evidence
•Adapter precedence documented for ELB/K8s/Consul
•Jittered schedules prevent probe storms

1	from dataclasses import dataclass
2
3	@dataclass
4	class ProbePolicy:
5	interval_s: int
6	timeout_ms: int
7	failure_threshold: int
8	success_threshold: int
9
10	def should_remove_from_lb(state: str) -> bool:
11	return state in {"UNHEALTHY", "UNKNOWN"}