Design a Recommendation ML Pipeline

Hard45 min
1 / 30
understanding•10 min read

Problem Statement: A Feedback-Driven Personalization Engine

Frames the system as a closed-loop data flywheel with three asymmetric workloads, not a lookup service.

Problem statement

Design a recommendation ML pipeline that ingests user event data (impressions, clicks, views, likes, purchases, dwell time), transforms those events into ML features, periodically retrains ranking models offline, and serves personalized recommendations in real time. The platform must also run experiments, track model versions, monitor drift, and roll back bad models without an outage.

This is not a CRUD backend and it is not a single model. It is a closed feedback loop: the system shows items, users react, reactions become labels, labels become training data, training data becomes a better model, and the better model changes what is shown. Every component sits on that loop. The moment you break the loop silently—by losing events, leaking labels, or serving stale features—recommendation quality decays without any error being thrown.

Three asymmetric workloads

  1. Event ingestion: extremely write-heavy, append-only, tolerant of eventual consistency but intolerant of silent loss. Assume hundreds of thousands of events per second at peak.
  2. Offline training: compute-heavy, batch, hours-scale, must be reproducible. It reads petabyte-scale history and writes gigabyte-scale artifacts.
  3. Online serving: extremely read-heavy and latency-critical. Tens of milliseconds p99, millions of requests per day per surface, and it must never block on the training plane.

A strong answer keeps these planes separate. Training may be six hours late; serving may never be sixty milliseconds late. The feature store is the contract that binds them: it must answer 'what did this feature look like at time T?' for training and 'what is this feature right now?' for serving, and the two answers must agree. That agreement problem—training/serving skew—is the defining engineering challenge of this design.

Public operating baseline versus design assumptions

Public evidence shows the category operates at extreme scale. Netflix reports that roughly 80% of content watched is discovered through its recommendation system and has described the recommender as worth more than $1B per year in retained subscriptions (Gomez-Uribe & Hunt, ACM TMIS 2015; Netflix Tech Blog). YouTube has stated that about 80% of watch time comes from recommendations and published its two-tower deep candidate-generation architecture at RecSys 2016. Pinterest reported 522M monthly active users in Q4 2024 and published PinSage, a graph neural network trained on a graph of 3B nodes and 18B edges (KDD 2018). These are cited company figures, not requirements for our fictional system.

For capacity planning, this answer explicitly assumes a mature product with 50M daily active users, 150M monthly active users, 1.2B recommendation requests per day, and 18B behavioral events per day with a 5x event peak. Unless a number is tied to a citation, it is a stated design assumption, target, or budget—not a claim about any company's private architecture.

The four architectural planes

  1. Event plane: collection, validation, deduplication, durable streaming, and archival of every interaction.
  2. Feature plane: batch and streaming feature computation, the online feature store, and point-in-time-correct training datasets.
  3. Model plane: training orchestration, evaluation gates, model registry, deployment, candidate retrieval, ranking, and re-ranking.
  4. Governance plane: experiments, guardrail metrics, drift monitoring, audit, and rollback authority.

Keep these planes separate. Let the model plane degrade without losing events, and let the governance plane halt a rollout without touching the serving hot path.

Key Highlights

  • •The system is a closed feedback loop: impressions create labels, labels create models, models create impressions.
  • •Three asymmetric workloads: write-heavy ingestion, compute-heavy training, latency-critical serving.
  • •Training may be hours late; serving may never be tens of milliseconds late.
  • •Training/serving skew—the feature store answering two different truths—is the defining engineering challenge.
  • •Four planes: event, feature, model, governance. Public figures are context; all scale numbers here are explicit assumptions.
Lead With the Train/Serve Split
State in the first two minutes that training latency is measured in hours and serving latency in tens of milliseconds, and that neither may block the other. This instantly distinguishes an ML pipeline design from a generic web backend.
Do Not Draw One Box That Does Everything
A single service that ingests events, trains models, and serves recommendations fails every workload simultaneously. Batch compute, stream processing, and low-latency inference have incompatible scaling and failure profiles.

Section Rescue Kit

Buzzwords to use:

Data FlywheelTraining/Serving Skew

Safe statements:

  • "Let me separate the three workloads first, because ingestion, training, and serving have completely different latency and consistency budgets."
  • "Before choosing tools, I want to name which decisions belong to the offline plane and which must be correct online without it."
Design a Recommendation ML Pipeline - System Design | WinJob | WinJob