Design a Recommendation A/B Testing Manager

Problem statement

Design a recommendation A/B testing manager: a platform that lets admins and data scientists create experiments comparing recommendation models or parameter sets, splits live traffic between variants at inference time, collects engagement metrics (impressions, clicks, conversions, dwell, watch time), and identifies a winner or stops underperforming variants early. The system must do this without spiking recommendation latency, without cross-contaminating user groups, and without rewriting history when configs change mid-flight.

This is not a generic web A/B tester. A recommendation experiment diverts traffic inside a serving path that may already produce thousands of candidates per second for hundreds of millions of users. The assignment decision happens on every recommendation request, the exposure (which variant actually produced which impression) must be logged immutably, and every downstream engagement event must be attributed to the exact variant version that caused it. Google's overlapping experiment infrastructure showed that running many simultaneous experiments requires orthogonal layers and domains so tests do not interfere [[2]]. LinkedIn publicly reports its experimentation engine serving up to 800,000 QPS of network calls and tens of thousands of simultaneous tests [[19]]. Those are cited company figures; our fictional system states its own assumptions explicitly.

Why the problem is distinctive

A checkout A/B test assigns once per user and measures one conversion. A recommendation experiment assigns on every request, produces billions of exposures, and measures a funnel of micro-conversions with heavy skew (a few users generate most of the engagement). Three properties dominate the design: (1) assignment must be deterministic, orthogonal across layers, and fast enough to live inside the ranking loop; (2) the exposure log, not the config, is the source of truth for attribution; (3) metric computation is a big-data pipeline problem (stream aggregation plus OLAP) with statistical rigor layered on top (SRM checks, variance reduction, sequential stopping).

The four architectural planes

1. Configuration plane: experiment definitions, layers, domains, allocations, metric definitions, guardrails; strongly consistent, versioned, signed propagation.
2. Serving plane: in-process assignment resolution inside the recommendation service, exposure logging, fallback to cached configs; never blocks ranking on the platform.
3. Measurement plane: event ingestion, stream aggregation, OLAP store, dashboards; freshness-bounded eventual consistency.
4. Analysis plane: statistical tests, CUPED, sequential boundaries, SRM detection, bandit controllers, decision workflow; batch and interactive.

A strong interview answer keeps these planes separate: the measurement plane may lag without harming serving, and the analysis plane may be offline without ever corrupting attribution.

Public operating baseline versus design assumptions

Public evidence establishes the category. LinkedIn describes serving up to 41,000 A/B tests simultaneously on its platform [[17]]. Microsoft's ExP lineage documented over 200 concurrent experiments per day at Bing as early as 2013 [[44]]. Netflix personalizes artwork with contextual bandits trained online and evaluated with inverse propensity scoring [[9]]. For capacity planning this answer explicitly assumes a mature product with 100M DAU, 4B recommendation impressions per day, 2,000 concurrent experiments, and a 5x event peak. Unless tied to a citation, every number is a stated assumption, target, or budget.