Design a High-Throughput Pub/Sub System

Problem Statement

Design a high-throughput publish/subscribe system that distributes events from thousands of producers to tens of thousands of consumers in near real time. Publishers write messages to named topics; multiple independent consumer groups each receive their own copy of every message, processing at their own pace. The system must sustain millions of messages per second while providing configurable delivery guarantees ranging from fire-and-forget to exactly-once.

This is not a point-to-point queue. The fundamental abstraction is a durable, partitioned, replayable log per topic. A message written once can be read by N consumer groups, each maintaining its own read position (offset), and re-read from any historical offset within the retention window. This decouples producers from consumers temporally and structurally: a producer does not know or care who consumes its events, and a consumer can join late and replay history.

Why This Problem Is Distinctive

A traditional message broker (RabbitMQ, ActiveMQ) routes individual messages to individual queues with immediate delivery. That model breaks at millions of messages per second because it couples the write path to the read path: if a consumer is slow, the broker buffers unboundedly or drops messages. A pub/sub system inverts this: the broker writes to an append-only log on disk and consumers pull at their own pace. Sequential disk I/O is orders of magnitude faster than random I/O, and the OS page cache serves recent reads for free.

The second distinctive challenge is fan-out at scale. A single topic might have 50 consumer groups. Writing the message once and letting each group read independently avoids the O(N) write amplification that a naive copy-per-consumer design would require. The log is the single source of truth; consumer groups are just independent cursors over it.

The third challenge is ordering. Global ordering across millions of messages per second is impossible without a single-writer bottleneck. The practical contract is per-partition ordering: messages with the same routing key land in the same partition and are delivered in write order within that partition. This is sufficient for most use cases (all events for user-123 go to partition 7 and arrive in order) while allowing massive parallelism across partitions.

The Four Architectural Planes

1. Ingest plane: accepts publishes, validates, batches, routes to partitions, and acknowledges.
2. Storage plane: durable, replicated, partitioned append-only logs with indexes.
3. Delivery plane: serves fetch requests to consumer groups, tracks offsets, manages rebalancing.
4. Control plane: metadata management, leader election, partition assignment, ACL enforcement, monitoring.

A strong interview answer keeps these planes separate. The storage plane can degrade (a broker goes down) without the control plane losing track of partition assignments, and the delivery plane can throttle a slow consumer without affecting the ingest plane's write path.

Public Operating Baseline

Apache Kafka, the canonical open-source implementation, was built at LinkedIn and published in 2011. LinkedIn reported processing 7 trillion messages per day across 100+ clusters in 2019, with peak throughput exceeding 2 million messages per second per cluster. Uber reported sustaining over 1 trillion messages per day through Kafka with their uReplicator multi-cluster tooling. Netflix's Keystone pipeline processes trillions of events per day through Kafka for real-time stream processing with Apache Flink. These are cited company figures, not requirements for our fictional system.

For capacity planning, this answer explicitly assumes a mature platform with 50,000 active publisher instances, 200,000 subscriber instances across 5,000 consumer groups, 5 million messages per second peak write throughput, and a 3× event peak multiplier. Unless a number is tied to a citation, it is a stated design assumption, target, budget, or illustrative threshold—not a claim about any company's private architecture.