Design a Recommendation Feedback Loop

Problem statement

Design a Recommendation Feedback Loop platform that captures every user reaction to a recommended item — click, skip, dwell, completion, rating, hide, report — and feeds those signals back into the recommendation pipeline so future suggestions measurably improve. The loop has five planes: capture (instrumented clients emit feedback events), transport (a durable event backbone moves billions of events per day), process (stream and batch jobs validate, join, and transform events into labels and features), learn (retraining and online-update pipelines turn labeled data into new models), and serve (the recommendation service consumes fresh models and features and emits scored results that close the loop).

A naive answer treats this as 'add a Kafka queue and a nightly retraining job.' The hard reality is that a feedback loop is a control system with delay, noise, and bias. Delay: a rating submitted today should improve tomorrow's ranking, and an implicit skip observed 200ms ago should update a real-time feature. Noise: bots, autoclicks, and accidental taps poison labels unless filtered. Bias: the loop only observes items the system chose to show, so a pure feedback-driven model collapses onto its own past — exploration, logging policies, and counterfactual correction are architecture requirements, not afterthoughts.

Why this question is distinctive

A generic analytics pipeline tolerates late and duplicated events; a feedback loop must preserve the join between the impression context (model version, experiment arm, candidate rank, features used) and the later outcome, because that join is the training label. If the context is lost, the label is unusable or, worse, misleading. This makes impression attribution a first-class data model, not a logging detail.

Public evidence shows the category operates at enormous scale. Netflix reports over 280 million paid memberships and has publicly stated that its recommendation system drives roughly 80% of content watched; Netflix also describes running hundreds of concurrent A/B tests. YouTube reports more than 2.5 billion logged-in monthly users and has published that roughly 70% of watch time comes from its recommendation system. LinkedIn reported crossing one billion members in November 2023. These are cited public figures; every capacity number later in this answer is an explicit design assumption.

The five planes

1. Capture plane: SDKs and servers emit typed feedback events with impression context, session, device, and monotonic sequence numbers.
2. Transport plane: partitioned, replayable event backbone with per-user ordering.
3. Process plane: stream processors for real-time features and fraud filters; batch jobs for label construction and dataset compaction.
4. Learning plane: feature store, training pipeline, model registry, evaluation gates, and experiment-aware promotion.
5. Serving plane: recommendation API that logs every scored impression with the exact model and arm that produced it.

A strong interview answer keeps the impression→outcome join at the center and treats retraining as one consumer of the loop, not the whole loop.