Problem Statement: One Control Plane, Hundreds of Clusters
Frames multi-cluster federation as a fleet-management control plane problem, not a bigger single cluster.
Problem statement
Design a multi-cluster Kubernetes federation: an overarching control plane that keeps deployments consistent across many Kubernetes clusters in different regions, clouds, and environments. Resources can be synced everywhere or selectively overridden per cluster group, with unified policy (RBAC, network policies, admission rules), aggregated observability, and automatic traffic distribution or cluster failover. The problem requires exactly those four capabilities: a federation controller that replicates and syncs resources, unified policy definitions, an aggregator for cluster-level metrics and logs, and automatic failover or traffic distribution.
Why one giant cluster is not the answer
The Kubernetes project's own cluster-size guidance caps a single cluster at roughly 5,000 nodes, 150,000 pods, and about 20 pods per node. Beyond the raw numbers, a single cluster is a single failure domain, a single upgrade train, a single etcd quorum, and a single blast radius. Real organizations split deliberately:
- Blast radius. A bad admission webhook, a misbehaving controller, or an etcd incident should take out one cluster, not the whole company.
- Data residency and compliance. EU personal-data workloads may legally run only in EU clusters; PCI workloads only in certified clusters. Location is a hard constraint, not a scheduling preference.
- Cloud and account limits. Provider quotas, IP space, and account boundaries force multiple clusters.
- Upgrade decoupling. 600 clusters cannot move in lockstep; waves per tier are mandatory.
- Team and tenant isolation. Noisy-neighbor contention at the control-plane API level is real at thousands of QPS.
Google learned this at the largest possible scale: the Borg paper (EuroSys 2015) describes cells of roughly 10,000 machines each, with workloads spanning many cells precisely because one cell is one failure and one upgrade domain. Kubernetes inherited the same conclusion: many clusters, one governance layer.
The four architectural planes
- Fleet control plane: source of truth for templates, placement, overrides, policy artifacts, traffic weights, and cluster membership. It writes intent; it never runs user workloads.
- Cluster agents: one small, pull-based agent per member cluster that applies assignments locally and reports status. The agent is the only component that talks to the member cluster's API server.
- Observability plane: fan-in for status, metrics, logs, and events with aggressive aggregation and cardinality control.
- Governance plane: policy-as-code, audit, release waves, and break-glass procedures.
The defining invariant
A federation control-plane outage must degrade management, never workloads. Member clusters keep running whatever they last received; agents keep reconciling local state; new rollouts simply pause. If your design makes workload liveness depend on the hub being reachable, it fails this interview immediately.
What 'consistent' means here
Consistency is eventual-by-design with bounded lag: a committed template change should be observable in every targeted cluster within seconds for small changes and within minutes fleet-wide during a storm, while per-cluster drift is detected within a fixed reconciliation window and either corrected or escalated according to an explicit conflict policy.
Key Highlights
- •Kubernetes guidance caps a single cluster near 5,000 nodes and 150,000 pods, so fleet-scale is inherently multi-cluster.
- •The control plane writes intent only; member clusters run workloads and must survive hub outage unchanged.
- •Data residency and compliance turn placement into a hard-constraint problem, not a scoring problem.
- •Four planes: fleet control plane, cluster agents, observability fan-in, governance.
- •Consistency means bounded eventual convergence plus explicit drift and conflict policy, not global locking.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "Before choosing any technology, let me separate who writes intent from who applies it and who observes it."
- "The design question is not how to make Kubernetes bigger, but how to govern many clusters as one fleet."