Problem Statement: A Distributed Build Farm for Monorepos
Frames the build farm as a correctness-first distributed system, not a pool of CI runners.
Problem statement
Design a distributed build farm that compiles, tests, and packages very large codebases — monorepos in the hundreds of millions of lines — by fanning work out to thousands of ephemeral build agents. The platform must accept build invocations from developers and CI, expand them into a dependency graph of individual actions, schedule those actions onto a worker fleet, cache repeated work in a content-addressed store, aggregate artifacts, and stream logs in near real time.
The trap in this question is drawing a bigger Jenkins. A build farm is not a queue of shell jobs on shared runners. It is a correctness system: if the cache ever returns a result produced from different inputs, different toolchains, or a polluted environment, engineers silently ship broken binaries and lose trust in the system permanently. So the design separates two concerns that weaker answers merge. Build progress is an eventually progressing workflow: queued, scheduled, fetching, executing, uploading, completed, failed. Build correctness is an invariant: an action may be served from cache only when its canonical key — command, input tree digest, toolchain digest, platform, and environment — matches exactly.
Why the problem is distinctive
A CI product can retry a flaky step. A build cache cannot un-poison a result that ten thousand engineers already downloaded. Large projects make this concrete. Google's 2016 CACM paper describes a single repository holding over one billion lines of code worked on by more than 25,000 engineers, with tens of thousands of changes landing per week; at that scale a clean build is measured in hours and the only viable strategy is aggressive caching plus remote execution. Meta open-sourced Buck in 2013 and Buck2 in 2023 with remote execution support; Twitter built Pants for a polyglot monorepo and shipped remote caching and execution as first-class features. The industry converged on a shared protocol — the Bazel Remote Execution API — which tells you the architecture shape is well understood and worth reproducing in an interview.
For capacity planning this answer assumes, and explicitly labels as assumptions: 25,000 engineers, 500,000 build invocations per day, 200 million actions per day, an 85% cache hit rate, a 4x peak multiplier, and a fleet of about 5,000 sixteen-core workers.
The five architectural planes
- Submission plane: authenticated APIs that accept invocations, enforce quotas, and return durable job handles.
- Graph plane: expands targets into an action DAG, deduplicates against the action cache, and pins exact toolchain versions.
- Scheduling plane: multi-tenant queues, priority aging, bin packing, leases, and preemption across worker cells.
- Execution and storage plane: ephemeral sandboxed workers, a content-addressed store for inputs and outputs, an action cache for results, and an artifact repository for publishable binaries.
- Observability plane: near-real-time logs, per-action traces, cache economics, and fleet health.
A strong answer keeps these planes separate. The scheduler may degrade and queue work without corrupting a single cached result, and the storage plane may shed telemetry load without ever serving a stale or wrong action result.
Key Highlights
- •A build farm is a correctness system first: a wrong cache hit is worse than a slow build.
- •Model build progress as a durable workflow and cache validity as a content-addressed invariant.
- •All uncited scale figures in this answer are explicit design assumptions, labeled as such.
- •Five planes: submission, graph, scheduling, execution/storage, observability.
- •The industry reference protocol is the Bazel Remote Execution API, used by Bazel, Buck2, and Pants.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate build progress, which may retry, from cache correctness, which must never be wrong."
- "Before drawing services, let me define what makes a cached result safe to reuse."