Design Video Analytics - System Design | WinJob

Problem Statement and Video Analytics Context

Design a planet-scale video analytics control plane that ingests playback beacons from every client surface (web, mobile, TV, embed) and powers creator dashboards, ops QoE war rooms, and experimentation—the product surface YouTube Studio, Mux Data, and Conviva sell. You are not designing the video player or CDN; you own telemetry correctness, cost-efficient rollups, and latency tiers per persona.

Problem framing

Anchor interviews on three consumers: creators who need watch time and retention curves; reliability engineers who need rebuffer and startup SLIs by CDN POP; data scientists who need cohort exports without PII leaks. State scale early: 500M DAU, ~50B beacons/day (~580k average events/sec, ~3M peak), p99 ingest ACK < 200ms, creator aggregates 1–5 minute lag acceptable.

Design choices

Fire-and-forget client beacons with disk-backed offline queue
Lambda architecture: stream path for ops alerts, batch path for billing-grade totals
Pre-aggregate in Flink/Spark before any dashboard hits OLAP

Deep dive

Separate QoE signals (startup_ms, rebuffer_ms, bitrate switches) from business events (view_start, complete, subscribe_click). Conviva-style session stitching merges beacons into a single playback timeline even when device_id rotates mid-session.

Operational detail

Mux Data proves developers pay for API-first metrics APIs—not only UI charts.
YouTube Studio hides multi-hour batch jobs behind “last updated 3h ago” labels for heavy reports.
Conviva sells real-time QoE to broadcasters where 30s alert latency saves ad slots.
Beacons are tiny JSON/protobuf—median 400B gzipped, 1.2KB uncompressed with rich device context.
Duplicate delivery is normal; idempotency keys are client-generated UUIDv7 per event.
Clock skew on mobile means server ingest_ts is authoritative for ordering, not client_ts alone.
GDPR erasure must cascade from user_id to session rollups within published SLA (72h typical).
Bot traffic is filtered at edge using device attestation + rate limits before Kafka spend.
Hot partitions appear on tentpole live streams—salting by session_id avoids skewed Kafka keys.
Sampling is acceptable for debug fields, never for revenue or SLA dashboards.
Cross-region ingest keeps users in-region; federated query layer merges for global executives.
Schema registry enforces backward-compatible Avro/Protobuf evolution for player SDK versions.
Dark launches ship new beacon fields behind feature flags with shadow validation pipelines.
Interviewers punish designs that scan raw S3 for dashboard queries—always mention rollups.
Out of scope: building ML recommender features, ad decisioning, or full BI semantic layer.
Success metric: player team trusts analytics enough to gate releases on QoE regressions.
Failure metric: beacon loss >0.1% or dashboard queries that time out on viral videos.
Cost metric: storage + compute per billion events—justify ClickHouse vs Druid vs BigQuery.

javaOne Dark Pro

1 public final class PlaybackBeacon {
2   private final String eventId;
3   private final String sessionId;
4   private final String videoId;
5   private final long clientTsMs;
6   private final int startupMs;
7   private final int rebufferMs;
8  
9   public PlaybackBeacon(String eventId, String sessionId, String videoId,
10       long clientTsMs, int startupMs, int rebufferMs) {
11     this.eventId = eventId;
12     this.sessionId = sessionId;
13     this.videoId = videoId;
14     this.clientTsMs = clientTsMs;
15     this.startupMs = startupMs;
16     this.rebufferMs = rebufferMs;
17   }
18  
19   public String eventId() { return eventId; }
20  
21   public boolean isDuplicateOf(PlaybackBeacon other) {
22     return eventId.equals(other.eventId);
23   }
24 }

pythonOne Dark Pro

1 from dataclasses import dataclass
2  
3 @dataclass(frozen=True)
4 class QoeSample:
5     session_id: str
6     startup_ms: int
7     rebuffer_ms: int
8     bitrate_kbps: int
9  
10 def rebuffer_ratio(sample: QoeSample, watch_ms: int) -> float:
11     if watch_ms <= 0:
12         return 0.0
13     return sample.rebuffer_ms / watch_ms

typescriptOne Dark Pro

1 interface AnalyticsBatch {
2   sessionId: string;
3   events: Array<{ eventId: string; type: string; ts: number }>;
4 }
5  
6 export function shouldFlush(
7   batch: AnalyticsBatch,
8   maxEvents: number,
9   maxAgeMs: number,
10   now: number,
11 ): boolean {
12   if (batch.events.length >= maxEvents) return true;
13   const oldest = batch.events[0]?.ts ?? now;
14   return now - oldest >= maxAgeMs;
15 }

Interviewer positioning

Open with personas, scale math, and explicit non-goals before drawing Kafka boxes.

Why interviewers care

Video Analytics interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for Problem Statement and Video Analytics Context that proves you understand real outages, not happy-path diagrams.

Key Highlights

•Fire-and-forget client beacons with disk-backed offline queue
•Lambda architecture: stream path for ops alerts, batch path for billing-grade totals
•Pre-aggregate in Flink/Spark before any dashboard hits OLAP

1	public final class PlaybackBeacon {
2	private final String eventId;
3	private final String sessionId;
4	private final String videoId;
5	private final long clientTsMs;
6	private final int startupMs;
7	private final int rebufferMs;
8
9	public PlaybackBeacon(String eventId, String sessionId, String videoId,
10	long clientTsMs, int startupMs, int rebufferMs) {
11	this.eventId = eventId;
12	this.sessionId = sessionId;
13	this.videoId = videoId;
14	this.clientTsMs = clientTsMs;
15	this.startupMs = startupMs;
16	this.rebufferMs = rebufferMs;
17	}
18
19	public String eventId() { return eventId; }
20
21	public boolean isDuplicateOf(PlaybackBeacon other) {
22	return eventId.equals(other.eventId);
23	}
24	}

1	from dataclasses import dataclass
2
3	@dataclass(frozen=True)
4	class QoeSample:
5	session_id: str
6	startup_ms: int
7	rebuffer_ms: int
8	bitrate_kbps: int
9
10	def rebuffer_ratio(sample: QoeSample, watch_ms: int) -> float:
11	if watch_ms <= 0:
12	return 0.0
13	return sample.rebuffer_ms / watch_ms

1	interface AnalyticsBatch {
2	sessionId: string;
3	events: Array<{ eventId: string; type: string; ts: number }>;
4	}
5
6	export function shouldFlush(
7	batch: AnalyticsBatch,
8	maxEvents: number,
9	maxAgeMs: number,
10	now: number,
11	): boolean {
12	if (batch.events.length >= maxEvents) return true;
13	const oldest = batch.events[0]?.ts ?? now;
14	return now - oldest >= maxAgeMs;
15	}