Design ML Content Moderation - System Design | WinJob

UGC Moderation Problem Framing

Multi-modal ML moderation is a publish-path safety system, not offline analytics. A mistaken removal triggers appeals, advertiser brand risk, and regulator scrutiny—while a mistaken allow surfaces harm at 2.4B items/day daily volume. This section hardens publish-path gate with measurable p99 text score < 120ms guardrails relevant to creator trust vs safety.

At 2.4B items/day scale, shard plans must assume creator skew—top creators generate bursts that crush naive per-item counters unless you pre-aggregate velocity features locally before Redis fan-in.

Operational reality: GPU pools preempt during viral events; design hold-not-remove degraded mode with rules-only + last-known-good vision models rather than silent allows.

For live surfaces, separate the sub-second comment lane from batch video transcode; mixing them in one queue destroys tail latency for creators mid-broadcast.

Quantified controls (UGC Moderation Problem Framing)

Control	Target	Why it matters
Primary SLI	p99 text score < 120ms	Anchors design reviews to observable outcomes
Volume signal	2.4B items/day	Drives partition counts and GPU fleet sizing
Risk focus	publish-path gate	Prevents hand-wavy ML talk without operations

Failure modes to pre-empt

Policy pack partial deploy across regions, embedding index lag after broker pause, adversarial evasion via obfuscated text, reviewer queue starvation when thresholds tighten without staffing, and label leakage when appeal outcomes join training without point-in-time guards.

Implementation slice

javaOne Dark Pro

1 public record ModerationDecision(
2     String decisionId,
3     String contentId,
4     String action,
5     double calibratedRisk,
6     String policyPackVersion,
7     String modelBundleHash
8 ) {}

pythonOne Dark Pro

1 from dataclasses import dataclass
2  
3 @dataclass(frozen=True)
4 class ModerationDecision:
5     decision_id: str
6     content_id: str
7     action: str  # allow|hold|remove
8     calibrated_risk: float
9     policy_pack_version: str
10     model_bundle_hash: str

typescriptOne Dark Pro

1 export interface ModerationDecision {
2   decisionId: string;
3   contentId: string;
4   action: "allow" | "hold" | "remove";
5   calibratedRisk: number;
6   policyPackVersion: string;
7   modelBundleHash: string;
8 }

Interview signal

Close by stating which SLI you instrument first—p99 text score < 120ms—and what architectural change you would make if it burned for seven days (e.g., widen review band, add locale-specific calibration, or shed video resolution on the hot path).

Why interviewers care

ML Content Moderation interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for UGC Moderation Problem Framing that proves you understand real outages, not happy-path diagrams.

Key Highlights

•Moderation focus: publish-path gate
•Scale signal: 2.4B items/day
•SLI anchor: p99 text score < 120ms

1	public record ModerationDecision(
2	String decisionId,
3	String contentId,
4	String action,
5	double calibratedRisk,
6	String policyPackVersion,
7	String modelBundleHash
8	) {}

1	from dataclasses import dataclass
2
3	@dataclass(frozen=True)
4	class ModerationDecision:
5	decision_id: str
6	content_id: str
7	action: str # allow\|hold\|remove
8	calibrated_risk: float
9	policy_pack_version: str
10	model_bundle_hash: str

1	export interface ModerationDecision {
2	decisionId: string;
3	contentId: string;
4	action: "allow" \| "hold" \| "remove";
5	calibratedRisk: number;
6	policyPackVersion: string;
7	modelBundleHash: string;
8	}