Problem Statement: Conversational LLM Product
Problem Statement: Conversational LLM Product — ChatGPT LLM serving interview depth
Problem Statement: Conversational LLM Product
ChatGPT is a multi-turn conversational product built on top of a large language model that streams its answer back one token at a time. At its core, the system takes the user's newest message together with the prior turns of the conversation, packs them into a single prompt that fits the model's context window, runs autoregressive inference across a fleet of GPUs, and pushes each generated token to the browser the moment it is produced.
The deceptively simple chat box hides a genuinely hard distributed-systems problem. A single GPU holds only a handful of in-flight conversations before its memory fills with attention state, yet the product must serve tens of millions of concurrent sessions. Each request is also long-lived: a 500-token answer streamed at 50 tokens per second occupies a GPU slot for ten full seconds, not the ten milliseconds a typical web request takes. That one fact inverts almost every assumption a stateless web backend is built on.
What we are actually building
We treat the product as four cooperating planes rather than a single model hidden behind an API. The control plane handles auth, rate limits, workspace policy, and billing; it decides whether and how a request runs. The context plane fetches conversation history, retrieves relevant documents, and packs everything into a fixed token budget. The inference plane is the GPU fleet running prefill and decode under a continuous-batching scheduler. The delivery plane streams tokens over Server-Sent Events and has to survive a client that disconnects in the middle of an answer.
The numbers that drive every decision
Quantify before naming hardware. At this scale we reason about roughly 20M concurrent sessions, on the order of 8B generated output tokens per day, and a target time-to-first-token (TTFT) under 800ms at p95. These three quantities — concurrency, tokens per second, and KV-cache footprint per session — are what set the partition keys, the GPU count, and the autoscaling signals. The specific model SKU is the last thing we choose, not the first.
Where it breaks
The real design pressure lives in the failure modes, not the happy path. A GPU can run out of memory mid-decode when too many long contexts land in the same batch. A user can close the tab after 300 of 500 already-billed tokens. A retried POST /messages can replay the same Idempotency-Key and double-charge the account. A moderation check can time out, forcing a choice between stalling the stream and shipping unvetted text. Each of these is something the interviewer will push on, and each has a concrete answer we build out in the sections that follow.
The interview frame: ChatGPT is far more than a wrapper around a model call. It is a backpressure-managed, GPU-scheduled streaming system in which cost, latency, and safety sit in permanent tension. Naming that tension out loud, with numbers attached, is the first signal a staff loop listens for.
Key Highlights
- •Model the product as four planes — control, context, inference, delivery — not a single model behind an API
- •Requests are long-lived: a 500-token reply at 50 tok/s holds a GPU slot for ~10s, breaking stateless-backend assumptions
- •Three numbers drive the design: ~20M concurrent sessions, ~8B output tokens/day, and TTFT under 800ms at p95
- •Lead with failure modes: GPU OOM mid-decode, disconnect after billed tokens, Idempotency-Key replay, moderation timeout
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I'll quantify concurrent sessions, tokens per second, and TTFT before naming any model SKU or GPU type."
- "Let me walk through client disconnect: we cancel the decode and free the KV cache so the GPU slot is reclaimed immediately."