Design a Monorepo CI Workflow

Medium45 min
1 / 30
System design answer

Design a Monorepo CI Workflow

Medium45 min30 sections
understanding•10 min read

Problem Statement: One Repository, Thousands of Modules, Finite CI Compute

Frames monorepo CI as a graph problem over change sets, not a pipeline problem over repositories.

Problem statement

Design a continuous integration workflow for a single Git repository that contains hundreds or thousands of modules: services, libraries, mobile apps, data jobs, and infrastructure definitions. Every commit and pull request must trigger CI that (1) detects which modules actually changed, (2) builds and tests only those modules plus the modules that depend on them, (3) caches partial results so unchanged work is never redone, (4) aggregates per-module results into one authoritative status, and (5) gates the merge based on that aggregated status.

The naive design rebuilds and retests the entire repository on every push. That design dies at scale: a full build of a multi-million-line monorepo can take hours and consume thousands of CPU-hours per run, while 95% of that work touches code the change never reached. The core engineering problem is therefore selective execution with correctness guarantees. A skipped test is only safe if you can prove the test result cannot change; the proof comes from the dependency graph and from hermetic, cacheable task definitions.

Why monorepo CI is distinctive

A multi-repo CI platform can treat each repository as an independent unit: one repo changed, one pipeline runs, one status returns. A monorepo destroys that assumption. A one-line change in a shared library can invalidate tests in three hundred downstream services, while a large refactor inside one leaf service invalidates nothing else. The same webhook event can require 4 tasks or 40,000 tasks depending on where the change lands in the dependency graph.

This forces three architectural separations:

  1. Change analysis is a graph computation, not a file diff. Path matching is a fast approximation; reverse-dependency queries over the module graph are the correctness argument.
  2. Build and test execution is a cache lookup first and a computation second. With content-addressed task keys, most tasks end in a cache hit, and the platform's job is to make hits cheap and trustworthy.
  3. Merge gating is a workflow over aggregated evidence. Individual task results mean nothing until they are rolled up into a single pass/fail decision attached to the exact commit under review.

Public operating baseline

Public evidence establishes that this category operates at extreme scale. Sadowski and the Google Piper team described in CACM in 2015 a single repository holding roughly two billion lines of code across nine million files and 86 TB, receiving around 60,000 commits per day including automated changes, with about 25,000 files changed per week. Microsoft's engineering blog described the Windows repository as roughly 3.5 million files and about 300 GB, which forced the invention of GVFS and later Scalar to make Git usable at that size. Meta open-sourced Buck2 in 2023 and described it as hermetic and materially faster than its predecessor, running on their monorepo with Sandcastle as the CI orchestrator. These are cited public figures for context, not requirements for our design.

For capacity planning this answer explicitly assumes a fictional mature monorepo: 2,400 engineers, 850 modules, 6.2 million lines of code, a 42 GB working tree, 3,200 pull requests per day, and an average of 5.4 commits per PR. Unless a number is tied to a citation, it is a stated design assumption.

The four architectural planes

  1. Analysis plane: change detection, module graph construction, reverse-dependency computation, and task graph synthesis.
  2. Execution plane: ephemeral runners, hermetic task execution, sharding, and remote build execution.
  3. Caching plane: content-addressed artifact storage, action cache, and layered local/remote caching.
  4. Governance plane: merge gating, status aggregation, flaky-test quarantine, metrics, and audit.

A strong answer keeps these planes separate: the merge gate must not depend on cache availability, the analysis plane must not require execution to answer 'what is affected', and cache misses must degrade to computation rather than to failure.

Key Highlights

  • •Monorepo CI is selective execution with correctness guarantees: skip work only when the dependency graph proves the result cannot change.
  • •Google's public CACM 2015 figures (2B lines, 9M files, 86 TB, ~60K commits/day) show the category operates at extreme scale; our fleet numbers are explicit assumptions.
  • •The design splits into analysis, execution, caching, and governance planes, each with independent failure behavior.
  • •A cache miss must degrade to recomputation, never to a failed build status.
  • •Merge gating consumes aggregated evidence for one exact commit, not a stream of partial task results.
Lead With the Graph
State in the first two minutes that monorepo CI is a graph problem: a change set plus a dependency graph produces the exact set of invalidated tasks. This instantly separates a real answer from a generic CI tutorial.
Do Not Draw a Rebuild-Everything Pipeline
A design that runs the full build and test suite for every commit is wrong by construction at monorepo scale; it wastes 90%+ of compute and makes feedback time unusable.

Section Rescue Kit

Buzzwords to use:

Selective ExecutionHermetic Task

Safe statements:

  • "I will separate change analysis from execution: analysis answers what is affected without running anything."
  • "Before choosing tools, let me define which decisions are graph-based proofs and which are path-based heuristics."
Design a Monorepo CI Workflow - System Design | WinJob | WinJob