Design GPU Scheduling

GPU Scheduling as a Cluster Operating System

Treat the scheduler as an OS for accelerators: admission, placement, preemption, and chargeback—not a cron wrapper around Kubernetes.

Google, AWS, and CoreWeave-style fleets treat GPU scheduling as the bottleneck between ML ambition and silicon budget. This section (1/30) anchors decisions interviewers expect you to defend with numbers, not framework names.

Mechanism 1A

Queueing interacts with gang allocation: a job requesting 64 GPUs stays queued until the scheduler can place all ranks on NVLink-aligned hosts, otherwise partial placement stalls the entire cluster with reserved-but-idle devices.

Mechanism 1B

Fair-share scheduling prevents a single team's 512-GPU pretraining run from starving 200 researchers waiting on 8-GPU fine-tuning jobs—deficit round-robin with priority caps is the usual compromise.

Mechanism 1C

Operability dashboards must show scheduling delay (time from submit to first GPU bound), placement failures, and preemption rate per pool; without them you cannot distinguish scheduler bugs from user mis-specified topology.

Interview sound bite

"If gpu scheduling as a cluster operating system fails, I first check whether we have partial gang deadlocks or quota borrow leaks before blaming NCCL."

Failure modes to volunteer

Head-of-line blocking, MIG oversubscription breaking training exclusivity, spot preemption without checkpoint hooks, and etcd leader flap pausing admissions while workers keep GPUs reserved.

Why interviewers care

GPU Scheduling interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for GPU Scheduling as a Cluster Operating System that proves you understand real outages, not happy-path diagrams.