Problem Statement: A Batch Supercomputer, Not a Web Service
Frames HPC as four planes - control, compute, fabric, storage - where exclusivity, topology and failure recovery dominate instead of request latency.
Problem statement
Design an on-premises High Performance Compute cluster that accepts batch jobs from thousands of researchers, schedules them onto thousands of CPU and GPU nodes, launches tightly coupled MPI jobs that span many nodes at once, feeds them data from a parallel file system, and keeps accurate accounting even though nodes, fabrics, and file-system targets fail regularly. The product must queue jobs with priority and fair share, gang-schedule parallel jobs so no rank waits, stage datasets, checkpoint long jobs, quarantine sick nodes, and bill accounts correctly.
This is the opposite shape of a web platform. A web service multiplexes many small requests onto shared capacity and optimizes p99 latency. An HPC cluster grants exclusive, topology-aware slices of hardware to one job at a time and optimizes aggregate scientific throughput measured in node-hours and FLOPS delivered. A scheduler mistake here does not return HTTP 500; it leaves 512 GPUs idle while a 48-hour job waits, or it lets one lab starve every other account for a week.
Why the problem is distinctive
Three properties separate HPC from every other DevOps design question. First, exclusivity and co-scheduling: an MPI job with 4,096 ranks either gets all 4,096 ranks running simultaneously or it makes near-zero progress, because collective operations block on the slowest rank. Second, topology is performance: placing two GPU ranks on different leaf switches can halve AllReduce bandwidth, so the scheduler must understand the fabric tree, not just count cores. Third, failure is routine at scale: Meta published in its OPT-175B training logbook roughly 181 interruptions across about 54 days of training, with GPU hardware failures the single largest cause category. A cluster that cannot detect a dead node in under a minute, requeue a job, and restart from the last checkpoint wastes more compute than it delivers.
Public evidence establishes the category scale. Frontier at Oak Ridge National Laboratory, the first exascale system on the TOP500 list, runs roughly 9,400 HPE Cray EX nodes with AMD EPYC CPUs and Instinct MI250X accelerators on an HPE Slingshot fabric, reports 1.194 exaFLOPS of HPL, and drew about 21 megawatts in its published Green500 submission. Fugaku at RIKEN runs 158,974 nodes. El Capitan at Lawrence Livermore reached 1.742 exaFLOPS HPL on the November 2024 TOP500 list. SchedMD reports that Slurm is the workload manager on a majority of TOP500 systems, which is why this design uses a Slurm-class architecture as its backbone.
Public operating baseline versus design assumptions
The national-lab figures above are context, not requirements for our system. This answer designs a fictional but realistic mid-size research cluster with 2,000 nodes and 4,800 GPUs, and every uncited number is an explicitly stated design assumption, budget, or target. Labeling assumptions is part of the interview discipline: a candidate who quotes exascale power figures as if they were the design target has not done capacity planning, they have done transcription.
The four architectural planes
- Control plane: the scheduler controller, node agents, accounting database, authentication, reservations, and policy (partitions, quality-of-service, fair share). This is the single-writer brain.
- Compute plane: CPU and GPU nodes with local NVMe, cgroups, NUMA binding, and a node daemon that launches, monitors, and kills tasks.
- Fabric plane: the high-speed interconnect - InfiniBand or Slingshot - with a topology the scheduler must understand, plus the management Ethernet.
- Storage plane: a Lustre-class parallel file system with separate metadata and data servers, an NVMe burst-buffer tier for checkpoint storms, and staging workflows.
A strong answer keeps these planes separate. Running jobs do not depend on the controller being alive - the node agents and the fabric keep them executing - the same way a data plane keeps serving when its control plane is down. The controller plans; the nodes execute; the file system feeds; the fabric connects. Each plane degrades differently and recovers differently, and the design must say so for each one.
Key Highlights
- •HPC optimizes aggregate node-hour throughput with exclusive, topology-aware allocations, not request latency.
- •MPI jobs need gang scheduling: all ranks start together or the job stalls on collective operations.
- •Meta's OPT-175B logbook published ~181 interruptions in ~54 days, with GPU failures the top hardware cause - failure handling is a primary requirement.
- •Frontier runs ~9,400 nodes at 1.194 exaFLOPS HPL and ~21 MW; SchedMD reports Slurm underpins a majority of TOP500 systems.
- •The architecture separates control, compute, fabric, and storage planes, each with its own degradation and recovery behavior.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "Let me separate what the scheduler plans from what the nodes execute, because running jobs must survive a controller outage."
- "Before choosing components, I will name the four planes and which one owns each failure mode."