Design Release Orchestration - System Design | WinJob

Problem Statement: Multi-Service Release Orchestration

Context

Spinnaker, Harness, and Argo CD (ApplicationSet + Rollouts) all solve multi-artifact, multi-environment promotion—but interviews want your orchestration layer: a release train with explicit DAG dependencies, human and automated gates, partial holds when one microservice fails, and a single dashboard for executives and on-call engineers.

Mechanisms you must articulate

Release plan — version pins per service, environment matrix (staging → prod-us → prod-eu), and ordered waves.
Gate types — manual approval, CI green, security scan, SLO comparison vs baseline, change-window calendar.
Failure semantics — fail-closed on missing telemetry; rollback wave vs hold downstream when an optional service fails.
Idempotency — every POST /releases/{id}/advance carries Idempotency-Key; adapters treat duplicate deploy requests as no-ops.

Numbers (state assumptions aloud)

800 microservices, 40 production deploys/day peak → ~0.5 deploy starts/sec average, burst 20/sec during coordinated trains.
Control plane stays < 200 QPS; heavy work is async workers + webhooks.
Audit log: ~2 KB/event × 50 steps/release × 40 releases/day ≈ 4 MB/day before compaction (trivial at Postgres scale).

Interview closer for this slice

Tie Problem Statement: Multi-Service Release Orchestration back to blast-radius: orchestration exists so no team ships an incompatible API version before its consumer’s DB migration completes.

javaOne Dark Pro

1 public final class ReleaseGateEvaluator {
2   public GateVerdict evaluate(StepRun run, GatePolicy policy) {
3     if (!run.allDependenciesSucceeded()) {
4       return GateVerdict.hold("upstream_failed");
5     }
6     if (run.sliBreaches(policy)) {
7       return GateVerdict.failClosed("sli_regression");
8     }
9     return GateVerdict.proceed();
10   }
11 }

pythonOne Dark Pro

1 def next_release_action(run: dict, policy: dict) -> str:
2     if not run.get("deps_ok"):
3         return "hold"
4     if run.get("error_rate", 0) - run.get("baseline_error", 0) > policy["max_delta"]:
5         return "rollback_wave"
6     return "advance" if run.get("manual_approved") else "wait_gate"

typescriptOne Dark Pro

1 export function advanceWave(
2   wave: { id: string; status: string },
3   gates: { passed: boolean }[],
4 ): "proceed" | "hold" | "rollback" {
5   if (gates.some((g) => !g.passed)) return "rollback";
6   if (wave.status === "blocked") return "hold";
7   return "proceed";
8 }

How to open this one

The framing that lands for multi-service release orchestration is coordinating dependent deploys as a DAG with gates, not firing them off independently: service B promotes only after service A's new contract is live, and a failure anywhere pauses the wave rather than leaving the fleet half-migrated. Lead with the dependency graph and the rollback-the-wave story — one service regresses mid-release and the orchestrator halts and reverts the whole coordinated set — and you have shown you understand orchestration is about ordering and atomicity across services.

Key Highlights

•Release orchestration limits blast radius by measuring candidate traffic before full promotion.
•Baseline comparison beats global averages when traffic is diurnal.
•Fail closed on stale metrics or failed mesh ACKs.
•Expand-contract migrations are mandatory under split traffic.

1	public final class ReleaseGateEvaluator {
2	public GateVerdict evaluate(StepRun run, GatePolicy policy) {
3	if (!run.allDependenciesSucceeded()) {
4	return GateVerdict.hold("upstream_failed");
5	}
6	if (run.sliBreaches(policy)) {
7	return GateVerdict.failClosed("sli_regression");
8	}
9	return GateVerdict.proceed();
10	}
11	}

1	def next_release_action(run: dict, policy: dict) -> str:
2	if not run.get("deps_ok"):
3	return "hold"
4	if run.get("error_rate", 0) - run.get("baseline_error", 0) > policy["max_delta"]:
5	return "rollback_wave"
6	return "advance" if run.get("manual_approved") else "wait_gate"

1	export function advanceWave(
2	wave: { id: string; status: string },
3	gates: { passed: boolean }[],
4	): "proceed" \| "hold" \| "rollback" {
5	if (gates.some((g) => !g.passed)) return "rollback";
6	if (wave.status === "blocked") return "hold";
7	return "proceed";
8	}