Design CI/CD Pipeline

Problem Statement: Enterprise CI/CD Platform

A CI/CD platform is the paved road every code change travels from a developer's laptop to running in production. Continuous Integration (CI) proves that each change still compiles and passes its tests the moment it lands on main; Continuous Delivery (CD) keeps that branch in a state that is always safe to ship. At the scale in this prompt — 500+ microservices, ~8,000 engineers, roughly 25,000 merges into main every week — the pipeline is not a developer convenience, it is the rate limiter on how fast the whole company ships. Google, Netflix, and GitHub do not grade this platform by which tools it stitches together; they grade it by two numbers: lead time from commit to production, and change failure rate, the fraction of deploys that trigger an incident.

What to state early

• 500+ microservices, 8k engineers, ~25k merges/week into main
• Target: p95 feedback on main < 12 min; prod deploys gated by policy + SLO
• Immutable artifacts promoted by digest, never by floating tags

The shape of the system

The platform splits cleanly into a control plane — which stores pipeline definitions, schedules runs, enforces policy, and records every promotion in an append-only ledger — and a fleet of stateless executors, the runners that actually compile, test, and package code. The control plane is the source of truth; runners are cattle, killable and replaceable mid-build without losing a run. Every box on the diagram has to earn its place by moving one of three numbers: the queue latency before a build starts, the stage success rate, and promotion correctness — the guarantee that only an artifact CI genuinely vetted can reach production.

When it degrades

The failure that matters here is a dependency stall — the artifact store, the SCM webhook ingress, or the runner pool itself. The correct reaction is to shed load gracefully: freeze new promotions, let in-flight runs drain, and keep one run timeline that on-call can read without grepping across three separate UIs. Recovery always rolls forward — redeploy the last known-good digest rather than rebuild from a tag that may have moved underneath you.

What it costs to run

The dominant operational cost is the runner fleet, and its enemy is idle capacity: autoscaling always lags a traffic spike, so a Monday-morning merge surge queues unless you keep warm headroom you are partly paying for around the clock. The second cost is the cache — a dependency upgrade invalidates it and triggers a cache-miss storm where every build recompiles from scratch, briefly multiplying build minutes. The operability lever for both is graceful load-shedding: when the fleet saturates, disable scheduled pipelines and defer non-default branches first, protecting default-branch feedback before you ever let overall availability slip.

CI versus CD, precisely

Continuous integration proves that every change still integrates — it builds and the test suite is green — within minutes of landing on main. Continuous delivery is the stronger discipline of keeping that branch releasable at all times, so shipping becomes a business decision rather than an engineering scramble. The practice underneath both is trunk-based development: short-lived branches that merge daily, which is what stops the merge queue from collapsing under 25,000 weekly merges. The trap to avoid is conflating delivery (always releasable) with deployment (actually released); separating them is what lets a regulated team park a signed build behind an approval gate for days without blocking everyone else. The strongest way to open this interview is to name the two metrics the platform exists to move — lead time and change failure rate — and to promise that any bad deploy is reversible by redeploying the last known-good digest in under two minutes.