Design a ML-Driven Content Moderation System (Text/Image/Video)

Problem statement

Design a content moderation platform that screens user-generated content across text, images, uploaded video, and live streams, detects policy violations using ML, routes ambiguous cases to human review, applies enforcement actions, and supports appeals. The system must operate at the speed of the feed: some violations must never be seen by another user, others can be caught by asynchronous sweeps within minutes, and live streams need takedown within seconds of detection.

This is not one classification problem. It is a portfolio of problems with different error tolerances. Zero-tolerance categories, such as known illegal material and terrorist content, demand near-perfect recall and justify fail-closed behavior and hash-based exact matching. Medium-severity categories, such as harassment or spam, tolerate a precision-recall trade-off and benefit from user trust tiers. Low-severity categories, such as borderline community-norm violations, should lean toward human review rather than automated removal, because the reputational and regulatory cost of a false removal is high.

Why the problem is distinctive

A recommendation system can ship a bad prediction and learn from clicks. Moderation cannot. First, the adversary adapts: bad actors use homoglyph substitution, leetspeak variants, steganography, cropped and re-encoded media, and coordinated evasion, so models degrade continuously and the system needs an evasion-detection and retraining loop. Second, the workload is multi-modal and latency-stratified: text decisions are expected inside the post round trip, images within a few hundred milliseconds, video over seconds to minutes, and live frames with a hard seconds-level takedown budget. Third, the system is legally exposed: regulations such as the EU Digital Services Act and the UK Online Safety Act require transparency reports, statement of reasons, and functioning appeal channels, so decisions must be explainable, versioned, and auditable.

Public operating baseline versus design assumptions

Public figures establish the scale of the category. YouTube reports more than 500 hours of video uploaded per minute, which is roughly 720,000 hours of new video per day. Meta reports about 3.1 billion daily active people across its Family of Apps and states that tens of thousands of people work on safety and security; its enforcement reports describe proactive detection rates above 99% for some visual violation categories. TikTok's transparency reports describe on the order of one hundred million videos removed per quarter, with a company-reported proactive removal rate above 95% and the large majority removed before receiving a single view. These are company-reported figures and context, not our requirements.

For capacity planning this answer explicitly assumes a mature platform with 500 million DAU, 2.5 billion content submissions per day (2.0 billion text, 400 million images, 100 million short videos of which 30 million are uploads processed in full), 200,000 concurrent live streams, and a 5x event peak. Every uncited number in this answer is a stated design assumption, target, or budget.

The four architectural planes

1. Screening plane: synchronous cheap filters, hash matching, and ML scoring that gate content at submission time.
2. Decision plane: the policy and rules engine that fuses scores with policy versions, jurisdiction, and user trust into an enforceable decision.
3. Enforcement and appeals plane: action execution, notifications, human review queues, appeals, and reinstatement.
4. Learning and governance plane: telemetry, ground truth from reviews and appeals, model evaluation, canary release, policy versioning, and compliance evidence.

A strong answer keeps these planes separate. The screening plane may degrade without blocking posting; the decision plane must stay strongly consistent per content item; the learning plane must never silently change live enforcement thresholds without a governed release.