Problem Statement: Coordination Is the Product, Not Builds
Frames multi-repo CI/CD as a graph-coordination and consistency problem, not a collection of independent build boxes.
Problem statement
Design a CI/CD orchestrator for an engineering organization whose code lives in many Git repositories. Repos depend on each other: service A imports library B, mobile app C consumes SDK D, and services E and F share proto contract G. When someone merges a change to B, the system must detect it, compute which downstream repos are affected, rebuild and retest the right set, publish versioned artifacts, and keep the whole estate in a coherent, deployable state. When many teams merge at once, the system must decide which combinations of commits are allowed to land together without breaking anyone else.
Single-repo CI is a solved-ish problem: one webhook, one pipeline, one artifact. Multi-repo CI is a distributed coordination problem wearing a DevOps costume. The hard questions are not "can we run make?" but: which builds are affected by this commit, which combination of repository SHAs is mutually consistent, which merges may proceed in parallel without breaking the trunk of any repo, and how do we cache enough work that the graph does not rebuild the world on every push.
Why this is distinctive
A wrong build in a single repo delays one team. A wrong decision in a multi-repo orchestrator can poison shared artifacts, break every consumer of a library, or let incompatible versions reach production together. The design therefore separates three concerns. Change detection is an ingestion and identity problem: webhooks arrive at-least-once, out of order, sometimes duplicated, and must be normalized into canonical commit events. Graph reasoning is a data problem: the dependency graph is versioned per manifest revision, and impact analysis must be computed against the graph as it existed at a given commit, not today's graph. Execution is a scheduling and correctness problem: builds run in topological waves, reuse cached artifacts by content hash, and gate merges through a serialized queue when the organization demands always-green trunks.
Google's public monorepo paper reports roughly two billion lines of code, nine million files, and about 60,000 commits per week served by one Piper repository and Blaze builds. Google largely eliminates this problem by removing the multi-repo boundary. Most companies cannot. Netflix publicly describes running over a thousand microservices, each in its own repository, coordinated by Spinnaker pipelines and shared tooling. That is the reality this design serves.
The four architectural planes
- Ingestion plane: webhook receivers, signature verification, event normalization, commit canonicalization.
- Graph plane: manifest parsing, versioned dependency graph, impact analysis, cycle detection.
- Orchestration plane: pipeline DAG execution, merge queue, release trains, priorities and quotas.
- Execution plane: agent pools, hermetic builds, remote artifact cache, artifact store, logs and evidence.
A strong answer keeps these planes separate. The graph plane can be wrong without builds running. The orchestration plane can be degraded without losing the ability to answer "what depends on what." And the execution plane must remain safe even when every control-plane component is briefly unreachable.
Key Highlights
- •The core product is cross-repo coordination: impact analysis, version alignment, and merge gating.
- •Dependency graphs are versioned per manifest revision; impact is computed at a commit, not against live state.
- •Google's monorepo (roughly 2B lines, 60K commits/week per Potvin and Leiserson) avoids this problem; multi-repo orgs must engineer it.
- •Four planes: ingestion, graph, orchestration, execution.
- •A wrong orchestrator decision can poison shared artifacts across every consumer repo.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate change detection, graph reasoning, orchestration, and execution before choosing any storage."
- "The interesting failures here are coordination failures, so let me define who owns version alignment first."