Design a Real-Time Analytics Pipeline

Problem statement

Design a real-time analytics pipeline that ingests high-velocity event data (clickstream, application logs, service metrics, transaction events), processes it in near real time with windowed aggregations and transformations, and serves dashboards and alerts with sub-second to sub-minute freshness. The system must survive burst traffic, tolerate late-arriving data, preserve correctness under restarts, and degrade predictably when any stage overloads.

This is not a database with a fast insert path. The defining constraint is the freshness contract: an event observed at the edge must influence an aggregate, an alert, or a dashboard value within a bounded time. Every layer—ingestion, transport, processing, serving, and alerting—is sized and evaluated against that contract.

Why the problem is distinctive

A batch warehouse can recompute yesterday. A real-time pipeline cannot recompute the last ten seconds without users noticing. Three properties make the problem hard:

1. Unbounded input. The stream never ends, so queries must be expressed over windows or materialized state, never over an unbounded scan.
2. Out-of-order delivery. Networks, retries, and device buffering mean an event for minute N can arrive during minute N+2. The pipeline must decide, with evidence, when a window is complete.
3. Failure during computation. A crash mid-window must not double-count or drop events. Recovery must be replayable and idempotent.

Public operating signals establish that this category runs at extreme scale. LinkedIn reports processing hundreds of billions of messages and hundreds of terabytes of data every day on Samza, its stream-processing platform built on Kafka, with hundreds of thousands of jobs across hundreds of clusters. Netflix reports that Apache Flink processes trillions of messages and petabytes of state per day in production. Cloudflare reports an HTTP request rate averaging about 47 million per second and peaking at roughly 86.5 million per second in early 2025. These are cited company figures used as context, not requirements for our fictional system.

The four architectural planes

1. Ingestion plane: collectors, edge buffering, schema validation, and broker write path.
2. Stream processing plane: stateful operators, windowing, watermarks, joins, and exactly-once recovery.
3. Serving plane: low-latency OLAP stores, real-time caches, WebSocket and SSE fan-out to dashboards.
4. Governance plane: schema registry, alert policy, lineage, access control, and quality monitoring.

A strong interview answer keeps these planes separate. Ingestion can degrade to buffered store-and-forward without corrupting serving. Serving can be stale without making processing wrong. The governance plane can reject a schema change without stopping in-flight events that use the old schema.

Public baseline versus design assumptions

All uncited numbers in this answer are explicit design assumptions. For capacity planning, this answer assumes a mature platform with 2 million events per second average ingestion, a 4× burst multiplier, 500-byte average event envelopes, and a freshness target of under 2 seconds event-to-dashboard for hot metrics and under 60 seconds for long-window aggregates.