Design a Data Pipeline with Kafka & Spark

Problem statement

Design a data pipeline in which Kafka is the ingestion backbone for high-throughput event data and Spark Structured Streaming performs transformations and aggregations, writing results to a data lake, a serving store, and real-time dashboards. The system must define a partitioning strategy, survive broker and processor failures, support checkpointing, and state exactly which delivery guarantee each edge provides.

The distinctive engineering fact of this design is that a stream pipeline has three different correctness surfaces. The ingestion surface must never lose acknowledged events and must allow replay. The processing surface must be restartable from a checkpoint without duplicating or dropping aggregates. The serving surface must expose freshness honestly so a dashboard never presents a stale window as current. A weak answer treats these as one pipe; a strong answer names the guarantee at each handoff.

Why Kafka plus Spark is a specific architecture, not a buzzword pair

Kafka gives you a durable, partitioned, replayable log with consumer-group parallelism and, since 0.11, transactions and idempotent producers. Spark Structured Streaming gives you a declarative incremental engine whose state is externalized to a checkpoint directory and whose micro-batch model turns stream correctness into a sequence of small, restartable batch commits. The combination is powerful precisely because Kafka is replayable: Spark can re-read any offset range after a failure, so recovery is replay-from-source plus checkpoint-restore, not fragile in-memory resurrection.

Public operating baseline versus design assumptions

Public evidence shows the category is production-real at extreme scale. Kafka was born at LinkedIn, which has publicly reported peaks above seven trillion messages per day and petabyte-scale daily ingress. Netflix's Keystone pipeline has been described as processing trillions of events per day over Kafka. Uber has described a data platform past one hundred petabytes with Kafka as the central event backbone. These are cited company figures for context, not requirements for our fictional system.

For capacity planning this answer explicitly assumes five billion events per day average, a five-times peak, and a one-kilobyte average serialized event. Unless a number is tied to a citation, it is a stated design assumption, target, budget, or illustrative threshold.

The four architectural planes

1. Ingestion plane: producers, schema registry, brokers, partitions, replication, retention.
2. Processing plane: Spark Structured Streaming jobs, checkpoint store, state store, watermarks, triggers.
3. Serving plane: data lake tables, serving store, dashboard cache, fan-out.
4. Governance plane: lineage, observability, security, retention, replay and rewind operations.

A strong interview answer keeps these planes separate, allows the serving plane to degrade without corrupting the lake, and never lets a dashboard SLO dictate ingestion durability.