Problem Statement: Web-Scale Search Ranking
Problem Statement: Web-Scale Search Ranking — search ranking interview depth
Problem Statement: Web-Scale Search Ranking
Search ranking decides, for a typed query against a corpus of billions of documents, which ten results land on the first screen — and it must do so in well under 200ms. At its core it is a two-stage retrieve-then-rank problem: a cheap, wide retrieval pass narrows billions of documents to a few hundred candidates, then an expensive L2 ranker (a neural model) scores those few hundred for relevance. You can never run the heavy model over the whole corpus, so that split is the whole game.
The systems that defined this — Google, Bing, Amazon product search — all optimize a measurable ranking metric, typically NDCG@10, against human judgments and click-derived labels. State the scale and the latency budget up front, because they constrain everything downstream: on the order of 500M queries/day, an index of ~10B documents, and a p95 near 180ms end to end, split across retrieve (~60ms), feature hydration (~40ms), and GPU scoring (~70ms).
The latency budget is the contract
Every later decision spends against that budget. Retrieval fans out across index shards in parallel and merges; feature fetch is a batched lookup keyed by doc_id; the L2 model scores in GPU batches. Log every impression before flushing the response bytes, or the training join between what was shown and what was clicked silently rots. On CAP, be specific per data type: AP for postings and click logs (a slightly stale shard beats a failed query — reconcile later), but stronger consistency for legal takedown flags, where serving removed content is never acceptable.
Where it breaks
The failure stories earn the credit: a viral query melts the single shard holding its hot term; a feature-store hot key forms on a trending doc_id; the ranker's GPU queue stalls under a latency spike; and a partial index replica serves stale results for minutes after a deploy. Name these, not a generic microservice diagram.
Key Highlights
- •two-stage retrieve-then-rank for billion-document corpora
- •learning-to-rank with click and dwell supervision
- •p95 query latency budget under 200ms end-to-end
- •freshness SLO for news and product inventory
NDCG@10 — the search-ranking quality metric
Normalized Discounted Cumulative Gain at 10: graded relevance, log-discounted by position, normalized against the ideal ordering so 1.0 is perfect.
1 import math 2 3 def ndcg_at_k(relevances: list[float], k: int = 10) -> float: 4 # relevances are graded labels in the order the model returned them 5 def dcg(rels: list[float]) -> float: 6 return sum((2 ** rel - 1) / math.log2(i + 2) for i, rel in enumerate(rels[:k])) 7 idcg = dcg(sorted(relevances, reverse=True)) 8 return dcg(relevances) / idcg if idcg else 0.0
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "For Problem Statement: Web-Scale Search Ranking, I will separate retrieve (wide recall) from L2 rank (narrow, expensive)."
- "Let me write the latency budget before picking Elasticsearch vs custom index."