Problem Statement: One Repository, Thousands of Modules, Finite CI Compute
Frames monorepo CI as a graph problem over change sets, not a pipeline problem over repositories.
Problem statement
Design a continuous integration workflow for a single Git repository that contains hundreds or thousands of modules: services, libraries, mobile apps, data jobs, and infrastructure definitions. Every commit and pull request must trigger CI that (1) detects which modules actually changed, (2) builds and tests only those modules plus the modules that depend on them, (3) caches partial results so unchanged work is never redone, (4) aggregates per-module results into one authoritative status, and (5) gates the merge based on that aggregated status.
The naive design rebuilds and retests the entire repository on every push. That design dies at scale: a full build of a multi-million-line monorepo can take hours and consume thousands of CPU-hours per run, while 95% of that work touches code the change never reached. The core engineering problem is therefore selective execution with correctness guarantees. A skipped test is only safe if you can prove the test result cannot change; the proof comes from the dependency graph and from hermetic, cacheable task definitions.
Why monorepo CI is distinctive
A multi-repo CI platform can treat each repository as an independent unit: one repo changed, one pipeline runs, one status returns. A monorepo destroys that assumption. A one-line change in a shared library can invalidate tests in three hundred downstream services, while a large refactor inside one leaf service invalidates nothing else. The same webhook event can require 4 tasks or 40,000 tasks depending on where the change lands in the dependency graph.
This forces three architectural separations:
- Change analysis is a graph computation, not a file diff. Path matching is a fast approximation; reverse-dependency queries over the module graph are the correctness argument.
- Build and test execution is a cache lookup first and a computation second. With content-addressed task keys, most tasks end in a cache hit, and the platform's job is to make hits cheap and trustworthy.
- Merge gating is a workflow over aggregated evidence. Individual task results mean nothing until they are rolled up into a single pass/fail decision attached to the exact commit under review.
Public operating baseline
Public evidence establishes that this category operates at extreme scale. Sadowski and the Google Piper team described in CACM in 2015 a single repository holding roughly two billion lines of code across nine million files and 86 TB, receiving around 60,000 commits per day including automated changes, with about 25,000 files changed per week. Microsoft's engineering blog described the Windows repository as roughly 3.5 million files and about 300 GB, which forced the invention of GVFS and later Scalar to make Git usable at that size. Meta open-sourced Buck2 in 2023 and described it as hermetic and materially faster than its predecessor, running on their monorepo with Sandcastle as the CI orchestrator. These are cited public figures for context, not requirements for our design.
For capacity planning this answer explicitly assumes a fictional mature monorepo: 2,400 engineers, 850 modules, 6.2 million lines of code, a 42 GB working tree, 3,200 pull requests per day, and an average of 5.4 commits per PR. Unless a number is tied to a citation, it is a stated design assumption.
The four architectural planes
- Analysis plane: change detection, module graph construction, reverse-dependency computation, and task graph synthesis.
- Execution plane: ephemeral runners, hermetic task execution, sharding, and remote build execution.
- Caching plane: content-addressed artifact storage, action cache, and layered local/remote caching.
- Governance plane: merge gating, status aggregation, flaky-test quarantine, metrics, and audit.
A strong answer keeps these planes separate: the merge gate must not depend on cache availability, the analysis plane must not require execution to answer 'what is affected', and cache misses must degrade to computation rather than to failure.
Key Highlights
- •Monorepo CI is selective execution with correctness guarantees: skip work only when the dependency graph proves the result cannot change.
- •Google's public CACM 2015 figures (2B lines, 9M files, 86 TB, ~60K commits/day) show the category operates at extreme scale; our fleet numbers are explicit assumptions.
- •The design splits into analysis, execution, caching, and governance planes, each with independent failure behavior.
- •A cache miss must degrade to recomputation, never to a failed build status.
- •Merge gating consumes aggregated evidence for one exact commit, not a stream of partial task results.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate change analysis from execution: analysis answers what is affected without running anything."
- "Before choosing tools, let me define which decisions are graph-based proofs and which are path-based heuristics."