Design Chaos Engineering Platform - System Design | WinJob

Problem Statement: Controlled Failure Injection at Scale

Why this matters for a chaos engineering platform

Interviewers at Netflix, Gremlin, and AWS expect you to connect experiment design, blast-radius policy, orchestration, and observability—not a box labeled "Chaos Monkey kills pods."

Core mechanisms

Experiment catalog — versioned scenarios (latency, CPU, packet loss, AZ drain, pod kill) with parameter schemas and rollback recipes.
Orchestrator — schedules runs, enforces approvals, coordinates agents, and records hypothesis → outcome.
Blast-radius governor — evaluates fleet percentage, tenant isolation, freeze windows, and dependency graphs before injection.
Observability hooks — pre/post dashboards, trace exemplars, and SLO burn gates tied to experiment IDs.

Operational invariants

Define steady-state hypotheses before touching production—availability, latency p99, and error rate must be measurable.
Production chaos requires executive policy — game days vs continuous low-intensity faults need different RBAC and paging rules.
Idempotent rollback — agents must restore iptables, cgroup limits, and routes even if the orchestrator crashes mid-run.

Interview signal

Quantify experiments/day, max concurrent blast radius (% of fleet), abort latency (<30s), and metadata retention (365d for regulated teams). Contrast Netflix Chaos Monkey lineage with Gremlin SaaS and AWS Fault Injection Simulator.

Phase focus (`understanding`)

Section 1 deepens understanding decisions for chaos platforms.

javaOne Dark Pro

1 // Illustrative: blast-radius gate (conceptual)
2 public final class BlastRadiusGate {
3   private final double maxFleetPercent;
4   public boolean allowRun(int targetedInstances, int fleetSize, boolean prod) {
5     if (prod && targetedInstances > fleetSize * maxFleetPercent) return false;
6     return targetedInstances > 0;
7   }
8 }

pythonOne Dark Pro

1 def allow_run(targeted: int, fleet: int, max_pct: float, prod: bool) -> bool:
2     if prod and targeted > fleet * max_pct:
3         return False
4     return targeted > 0

typescriptOne Dark Pro

1 export function allowRun(
2   targeted: number,
3   fleet: number,
4   maxPct: number,
5   prod: boolean,
6 ): boolean {
7   if (prod && targeted > fleet * maxPct) return false;
8   return targeted > 0;
9 }

How to open this one

The framing that lands for a chaos platform is the controlled experiment: a hypothesis, a bounded blast radius, automated abort criteria, and observability to confirm or refute — not "Chaos Monkey kills pods." Lead with the blast-radius governor (fleet percentage, tenant isolation, freeze windows) and the failure story that proves it: an experiment's abort condition trips and the platform halts and restores faster than the damage can spread. That shows you understand chaos engineering is safety engineering, not randomness.

Key Highlights

•Steady-state hypotheses precede fault injection
•Blast-radius governor caps concurrent production impact
•Orchestrator ≠ agent — central policy, distributed execution
•Auto-abort ties chaos runs to SLO/error-budget gates

1	// Illustrative: blast-radius gate (conceptual)
2	public final class BlastRadiusGate {
3	private final double maxFleetPercent;
4	public boolean allowRun(int targetedInstances, int fleetSize, boolean prod) {
5	if (prod && targetedInstances > fleetSize * maxFleetPercent) return false;
6	return targetedInstances > 0;
7	}
8	}

1	def allow_run(targeted: int, fleet: int, max_pct: float, prod: bool) -> bool:
2	if prod and targeted > fleet * max_pct:
3	return False
4	return targeted > 0

1	export function allowRun(
2	targeted: number,
3	fleet: number,
4	maxPct: number,
5	prod: boolean,
6	): boolean {
7	if (prod && targeted > fleet * maxPct) return false;
8	return targeted > 0;
9	}