Problem Statement: Containerizing HPC Without Losing Bare-Metal Performance
Frames the core tension between HPC's bare-metal performance requirements and containerization's deployment uniformity.
Problem Statement
Design a container-based virtualization platform for High-Performance Computing (HPC) workloads. Each HPC job runs inside a container, orchestrated by a specialized HPC scheduler such as Slurm, PBS Professional, or IBM Spectrum LSF. The platform must support GPU pass-through for GPGPU tasks, MPI across containers for parallel jobs, high-speed InfiniBand networking, and ephemeral container creation and teardown per job.
HPC has historically run bare metal. Scientists and engineers compile against specific MPI implementations, link against vendor-tuned BLAS libraries, and depend on exact CUDA driver versions. A single library mismatch can produce silent numerical errors or segfaults across thousands of nodes. Containerization solves this environment-conflict problem, but introduces a new one: any overhead from the container runtime, filesystem layer, or device abstraction directly degrades the performance that HPC users pay for.
Why This Problem Is Distinctive
A web-service container can tolerate 5% overhead. An HPC container cannot. The Frontier supercomputer at ORNL delivers 1.102 exaflops across 9,408 AMD EPYC CPUs and 37,632 AMD MI250X GPUs. A 2% overhead on a 12-hour simulation running on 8,000 GPUs wastes approximately 1,536 GPU-hours, worth roughly $4,600 at typical HPC cloud rates. The containerization layer must be effectively invisible to the compute path while still providing isolation, reproducibility, and dependency management.
The design must bridge two worlds that evolved independently. The container ecosystem (Docker, OCI images, registries) was built for microservices: short-lived, network-attached, root-privileged, overlay-filesystem workloads. The HPC ecosystem (Slurm, MPI, InfiniBand, Lustre) was built for batch jobs: long-running, tightly-coupled, unprivileged, shared-filesystem workloads. Neither was designed with the other in mind.
The Four Core Requirements From the Brief
- Container images with HPC libraries (MPI, CUDA): Images must carry OpenMPI or MPICH, CUDA toolkit or ROCm, and domain-specific libraries (FFTW, HDF5, PETSc) while remaining compatible with host-level MPI and GPU drivers.
- HPC scheduler integration for parallel jobs: The container runtime must integrate with Slurm, PBS Pro, or LSF so that multi-node MPI jobs launch containers on all allocated nodes simultaneously, with correct environment propagation.
- GPU pass-through for GPGPU tasks: NVIDIA GPUs must be visible inside containers via the NVIDIA Container Toolkit or equivalent, with near-native CUDA performance. AMD GPUs require ROCm pass-through.
- Ephemeral container creation and teardown: Containers are created per-job and destroyed after completion. No long-running daemons on compute nodes. Image caching must be fast to avoid I/O bottlenecks at job start.
Public Operating Baseline
Public evidence establishes that HPC containerization is operationally real. NERSC runs Shifter on Perlmutter, serving over 7,000 users. ORNL uses Apptainer (formerly Singularity) on Frontier and Summit. NVIDIA publishes Enroot and Pyxis for Slurm-integrated container execution. Los Alamos National Lab developed Charliecloud for unprivileged containers. These are production systems, not research prototypes.
For capacity planning, this answer assumes a mid-size national lab cluster: 2,000 compute nodes, 8,000 GPUs (4 per node), 200 Gbps HDR InfiniBand per node, 50,000 jobs per day, and a 3x peak multiplier. Unless a number is tied to a published source, it is a stated design assumption.
The Four Architectural Planes
- Compute plane: Container runtime, GPU pass-through, InfiniBand device access, cgroup enforcement.
- Scheduling plane: Slurm or PBS integration, job allocation, container launch orchestration, MPI bootstrap.
- Image plane: Container registry, image caching, layer management, signing and verification.
- Data plane: Shared filesystem (Lustre, GPFS), job I/O, checkpoint/restart, scratch storage.
A strong interview answer keeps these planes separate. The scheduling plane can degrade without affecting running containers. The image plane can be cached locally to survive registry outages. The data plane must never be blocked by container lifecycle operations.
Key Highlights
- •HPC containerization must add less than 2% overhead to compute-bound workloads to be viable.
- •The Frontier supercomputer at ORNL delivers 1.102 exaflops across 9,408 CPUs and 37,632 GPUs.
- •Container images must carry MPI, CUDA/ROCm, and domain libraries while remaining compatible with host drivers.
- •The design bridges two ecosystems that evolved independently: OCI containers and HPC batch scheduling.
- •Four architectural planes: compute, scheduling, image, and data. Each degrades independently.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate job scheduling from container execution: the scheduler allocates resources, the runtime executes the workload."
- "Before selecting technologies, let me define which operations run on the compute node, which run on the login node, and which run on the scheduler."