Design a Content Moderation Pipeline

Problem statement

Design a pipeline that ingests user-generated content (text, images, video), scans it with rules and ML models for policy violations such as hate speech, adult content, spam, harassment, self-harm, and violent content, quarantines suspicious items for human review when confidence is borderline, enforces decisions with an auditable ladder, and supports appeals. The brief requires four capabilities: UGC ingestion, ML or rule-based scanning, a moderator approve/reject workflow, and decision logging for auditing.

The distinctive engineering insight is that moderation is a decision factory with heterogeneous latency budgets, not a single filter. A synchronous gate must decide in tens of milliseconds whether obviously violating content may publish at all; an asynchronous cascade of heavier models re-scores everything within seconds; a human review queue absorbs the borderline band; and an enforcement engine translates decisions into proportionate, reversible actions. Every plane has different consistency, retention, and failure semantics, and a strong answer keeps them separate.

Why the split matters

If every check ran inline, publish latency would equal the slowest model and a model outage would halt posting. If nothing ran inline, the first minutes of a viral violation become unavoidable harm. Production systems therefore place a cheap, deterministic gate (externally-managed denylists, hash matches, a distilled model) in the publish path, and defer expensive vision, language, and cross-modal models to an async path that can still remove content within seconds and queue borderline items for reviewers.

Public operating baseline versus design assumptions

Company transparency reporting shows the category is real at extreme scale. YouTube reports removing on the order of ten million videos per quarter for policy violations, with the large majority flagged by automated systems before any human report, and most removals completed within hours. Meta reports proactive detection rates near or above 99% for prioritized categories such as terrorist propaganda and adult nudity, and has publicly disclosed a content review workforce in the tens of thousands. Discord publishes Automod and safety tooling built on streaming pipelines. These are cited, company-reported figures; they are context, not requirements for our fictional platform.

For capacity planning this answer explicitly assumes a mature platform with 250 million DAU, 120 million UGC submissions per day (84 million text, 30 million images, 6 million videos), 2.5 million user reports per day, and a 3x peak multiplier. Unless a number is tied to a citation, it is a stated design assumption, target, or budget.

The four architectural planes

1. Ingestion plane: upload acceptance, media normalization, metadata extraction, deduplication, and event publication with idempotency.
2. Automated decision plane: synchronous gate plus asynchronous model cascade, hash matching, score fusion, and a policy engine that maps scores to actions under versioned rules.
3. Human review plane: prioritized queues, leased case assignment, consensus and quality control, SLA timers, and reviewer wellbeing guardrails.
4. Enforcement and learning plane: the enforcement ladder, notifications, appeals, audit ledger, takedown fan-out, and the feedback loop that turns reviewed decisions into training and evaluation data.

A strong interview answer states in the first two minutes that the pipeline optimizes three competing currencies: harm minimized (violating exposure seconds), speech protected (false positives and appeal overturn rate), and cost bounded (compute per item and reviewer minutes).