Design ChatGPT

Problem statement

Design ChatGPT: a conversational product backed by a large language model (LLM) that accepts a user prompt, streams generated tokens back in real time, retains multi-turn conversation context, and scales to hundreds of millions of weekly active users. The system must host a frontier model (GPT-4 class or larger), schedule GPU/TPU inference across a large fleet, keep first-token and inter-token latency low, and survive node failures mid-generation without losing the user's session.

This is not a CRUD web app. A classic REST service can scale horizontally on cheap CPU pods; an LLM serving platform is GPU-bound, memory-bound, and latency-sensitive at the token level. A single H100 costs roughly $30,000 and can serve only a limited number of concurrent sequences before running out of high-bandwidth memory. The scarce resource is not requests-per-second on a load balancer — it is GPU hours and KV-cache bytes. Every architecture decision ultimately answers: how do we extract maximum useful throughput from each accelerator while keeping tail latency acceptable?

OpenAI launched ChatGPT on November 30, 2022. It reached 100 million monthly active users by January 2023, making it the fastest-growing consumer application at the time (per UBS and Sensor Tower analyses). By late 2025, OpenAI reported roughly 800 million weekly active users across its products. ChatGPT runs on Azure supercomputers built for OpenAI, each comprising tens of thousands of NVIDIA A100 and later H100 GPUs. These are public, self-reported figures; our design uses them only as scale context and labels all uncited numbers as explicit assumptions.

Why this problem is distinctive

A normal web request is a stateless request-response pair with millisecond DB reads. LLM inference is a stateful, autoregressive, GPU-resident loop: each generated token depends on every previous token, and the intermediate key-value (KV) attention cache must stay in fast GPU memory for the life of the sequence. A 4K-context conversation can consume hundreds of megabytes of KV cache per request. If we naively pin one GPU to one request, utilization collapses and cost explodes. If we overbatch, tail latency and out-of-memory failures spike. The core engineering tension is therefore throughput versus latency versus memory, managed by continuous batching, paged KV-cache allocation, and prefix caching.

The attached brief requires GPU/TPU-backed inference, chat session state and context retention, real-time token streaming, optional per-user or per-org fine-tuning, scalability across many GPUs, low-latency token generation, reliability with mid-inference fallback, and cost optimization. The section rhythm, phase vocabulary, cloud diagrams, savior system, recap contract, and WinSystemDesign Friend rules follow the course profile and master prompt. All three of the hardest subsystems — the batching scheduler, the KV-cache manager, and the streaming edge — get dedicated deep dives.

Public operating baseline versus design assumptions

Public evidence establishes that the category is operationally real. OpenAI has described running on custom Azure supercomputers; Microsoft's Azure OpenAI Service offers provisioned throughput units (PTUs) as a capacity model; NVIDIA publishes TensorRT-LLM with in-flight batching; and the open-source vLLM project from UC Berkeley introduced PagedAttention, reporting up to 24x throughput over a naive baseline. These are cited, self-reported figures and are used for context only.

For capacity planning, this answer assumes a mature platform with 200 million weekly active users, 20 million daily active chat sessions, and a peak of 400,000 concurrent active generations, with a five-times event peak multiplier. Unless a number is tied to a citation, it is a stated design assumption, target, budget, or illustrative threshold — not a claim about any company's private architecture.

The four architectural planes

1. Edge & session plane: API gateway, authentication, rate limiting, WebSocket/SSE streaming connections, conversation history storage.
2. Inference control plane: request queue, batching scheduler, KV-cache allocator, model router, prefix cache.
3. Accelerator plane: GPU/TPU worker pool running the model, continuous batching loops, health monitoring, failure detection.
4. Learning & governance plane: telemetry, evaluation, moderation pipeline, feature/personalization store, model registry, release gates.

A strong interview answer keeps these planes separate. It lets the session plane degrade without dropping in-flight tokens, and it lets the governance plane evaluate models without silently changing the validated serving path.