Design Immutable Infrastructure - System Design | WinJob

Immutable Infrastructure Interview Framing

Define immutability as rebuild-and-replace instead of SSH patching; cite Netflix Spinnaker AMI pipelines and Google Borg cell replacement as reference patterns.

When discussing cattle-not-pets, anchor every decision to the invariant that production instances are read-only. Changes flow through a new golden artifact, validated promotion gates, and a replacement controller that never mutates disk on running hosts.

Numbers to state early

Fleet scale: ~18,000 VMs/containers managed as cattle across 6 regions
Deploy cadence: 40–120 replacement waves per day with median 8-minute wave duration
Artifact registry: 3-year retention for prod digests; dev images GC after 14 days
Failure budget: halt wave if 5xx rate exceeds 2× baseline for 5 minutes

Mechanism deep dive

For immutable infrastructure interview framing, the control plane records an immutable audit event before any instance launches. Controllers compare desired generation (digest sha256:…) with live generation tags on each instance. Mismatch triggers replace workflow—not Ansible playbooks.

Operational edge cases

Long-running TCP sessions require drain hooks; batch jobs need graceful SIGTERM windows; stateful tiers use rack-aware replacement. Emergency break-glass launches a new image with hotfix baked in—still no SSH patch.

Comparison table

Approach	Drift risk	Rollback speed	Audit clarity
SSH patch	High	Slow, unclear	Poor
Config management on running host	Medium	Medium	Partial
Immutable replace	Low	Fast (redeploy N-1)	Excellent

javaOne Dark Pro

1 public final class ImagePromotionGate {
2   public boolean promote(boolean signed, boolean cvePass, boolean sbomPresent) {
3     return signed && cvePass && sbomPresent;
4   }
5 }

pythonOne Dark Pro

1 from dataclasses import dataclass
2  
3 @dataclass(frozen=True)
4 class GoldenArtifact:
5     digest: str
6     build_id: str
7     channel: str
8  
9 def can_roll_fleet(old: GoldenArtifact, new: GoldenArtifact, min_healthy: float) -> bool:
10     return new.channel == "prod" and new.digest != old.digest and min_healthy >= 0.9

typescriptOne Dark Pro

1 interface ReplacementWave {
2   percent: number;
3   minHealthy: number;
4   maxParallel: number;
5 }
6  
7 export function scheduleWave(currentPct: number, wave: ReplacementWave): number {
8   const step = Math.min(wave.percent, 100 - currentPct);
9   return currentPct + step;
10 }

cattle-not-pets takeaway

State aloud how cattle-not-pets reduces MTTR: operators swap artifact IDs, not debugging snowflakes. Interviewers at Netflix/Google/HashiCorp expect you to connect this section to fleet-wide consistency and supply-chain attestations.

How to open this one

The framing that signals depth on immutable infrastructure is replace-don't-patch: servers are never modified in place; a change means baking a new image and rolling it out, so every running instance is a known, reproducible artifact. Lead with why this kills configuration drift — the classic "works on that one node" outage — and the failure story that proves it: a bad image rolls out, and because nothing was mutated, rollback is just redeploying the previous image. That shows you understand immutability buys reproducibility and clean rollback.

Key Highlights

•Section 1 reinforces read-only runtime for cattle-not-pets
•Promotion gates block unsigned or vulnerable artifacts
•Wave orchestration couples drain, health, and terminate
•Audit lineage links git commit to live instance digest

1	public final class ImagePromotionGate {
2	public boolean promote(boolean signed, boolean cvePass, boolean sbomPresent) {
3	return signed && cvePass && sbomPresent;
4	}
5	}

1	from dataclasses import dataclass
2
3	@dataclass(frozen=True)
4	class GoldenArtifact:
5	digest: str
6	build_id: str
7	channel: str
8
9	def can_roll_fleet(old: GoldenArtifact, new: GoldenArtifact, min_healthy: float) -> bool:
10	return new.channel == "prod" and new.digest != old.digest and min_healthy >= 0.9

1	interface ReplacementWave {
2	percent: number;
3	minHealthy: number;
4	maxParallel: number;
5	}
6
7	export function scheduleWave(currentPct: number, wave: ReplacementWave): number {
8	const step = Math.min(wave.percent, 100 - currentPct);
9	return currentPct + step;
10	}