Design LLM Serving

Hard45 min
1 / 30
understanding•9 min read

Problem Statement: Hyperscale LLM Inference Platform

Problem Statement: Hyperscale LLM Inference Platform — LLM serving interview depth

Problem Statement: Hyperscale LLM Inference Platform

Design a hyperscale LLM inference platform (chat, embeddings, batch) with GPU fleets, streaming APIs, and safe rollouts.

Why interviewers probe here

OpenAI, Google, and Anthropic interviews test whether you treat LLM inference as a token factory with strict SLOs—not a single GPU running Hugging Face.

  • Control plane: model registry, routing, safety policy, rollout gates
  • Data plane: GPU workers with continuous batching and KV-cache reuse
  • Clients: chat completions, embeddings, batch JSONL, SSE token streams
  • Observability: TTFT, tokens/sec, GPU memory, queue depth, $/1M tokens

Operational detail

State the contract: prompts are immutable requests; responses stream tokens; model weights are versioned artifacts loaded once per worker.

Failure and edge cases

KV-cache eviction storms, prefill queue blowups, weight download blocking cold pods, and context-length abuse exhausting VRAM.

Interview checkpoints

  • Separate training from serving hot path
  • Name personas—API consumer, safety reviewer, GPU SRE

Implementation sketch

javaOne Dark Pro
1 
2public record LlmRoute(String modelId, String version, String stage) {}
3
pythonOne Dark Pro
1 
2@dataclass(frozen=True)
3class LlmRoute:
4    model_id: str
5    version: str
6    stage: str  # prod|canary|shadow
7
typescriptOne Dark Pro
1 
2export interface LlmRoute { modelId: string; version: string; stage: "prod" | "canary" | "shadow"; }
3

Interview signal

Frame LLM serving as a token factory with strict SLOs — TTFT, tokens/sec, and $/1M tokens — not a single GPU running Hugging Face. The tell is separating the control plane (registry, routing, safety, rollout gates) from the GPU data plane, and naming a real failure like a KV-cache eviction storm rather than a happy-path diagram.

Key Highlights

  • •Control plane: model registry, routing, safety policy, rollout gates
  • •Data plane: GPU workers with continuous batching and KV-cache reuse
  • •Clients: chat completions, embeddings, batch JSONL, SSE token streams
  • •Observability: TTFT, tokens/sec, GPU memory, queue depth, $/1M tokens
Key insight
OpenAI, Google, and Anthropic interviews test whether you treat LLM inference as a **token factory** with strict SLOs—not a single GPU running Hugging Face.
Avoid
KV-cache eviction storms, prefill queue blowups, weight download blocking cold pods, and context-length abuse exhausting VRAM.

Section Rescue Kit

Buzzwords to use:

Continuous BatchingPagedAttention

Safe statements:

  • "For Problem Statement: Hyperscale LLM Inference Platform, I separate admission control from GPU scheduling and quote token throughput, not vague QPS."
  • "I'll state TTFT and $/1M tokens SLIs before picking H100 counts or TP degree."
Design LLM Serving - System Design | WinJob | WinJob