Design Object Detection - System Design | WinJob

What we are designing

Design a multi-tenant object detection platform that ingests camera frames (warehouse robots, retail shelves, traffic cameras, in-vehicle ADAS feeds) and returns axis-aligned or rotated bounding boxes with class labels and confidence scores. Tesla interviews stress latency under motion blur; Google Cloud Vision and Amazon Rekognition expose DetectLabels/DetectCustomLabels with box geometry; warehouse automation teams need sub-200ms p99 on 1080p frames.

Core user journeys

Fleet operator: registers cameras → defines class taxonomy (COCO-80 vs custom SKUs) → sets score/NMS thresholds → streams RTSP/WebRTC frames → consumes JSON boxes over webhook or polls job status for batch MP4.

ML engineer: uploads ONNX/TorchScript detector → runs offline eval on holdout set (mAP@0.5:0.95) → promotes canary route 5% → monitors precision/recall per class.

Why interviewers ask this

Object detection couples computer vision with distributed streaming: decode farms, GPU batching, NMS on GPU, tracking across frames (IoU association), and safety policies when a missed pedestrian box has legal consequences.

Scale anchors (state explicitly)

900M frames/day analyzed (mix of 15fps traffic + 30fps retail) → ~10,400 average frame QPS, 83,000 peak with 8× factor
1080p JPEG ~280KB after edge downscale; 4K bursts for two tenants only
120 detector variants (YOLOv8, Faster R-CNN, custom retail heads) on 1,400 A100-class GPUs across 5 regions

Architecture split

Control plane: model registry, per-tenant route table, class ontology, threshold policies, billing meters. Data plane: ingest → decode/letterbox → GPU infer → NMS + coordinate remap → optional tracker → durable detection log + live websocket fanout.

javaOne Dark Pro

1 public record DetectRequest(String tenantId, String frameUri, String modelId, float scoreThreshold) {}

pythonOne Dark Pro

1 @dataclass(frozen=True)
2 class DetectRequest:
3     tenant_id: str
4     frame_uri: str
5     model_id: str
6     score_threshold: float

typescriptOne Dark Pro

1 export interface DetectRequest {
2   tenantId: string;
3   frameUri: string;
4   modelId: string;
5   scoreThreshold: number;
6 }

Why interviewers care

Object Detection interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for Problem Statement: Real-Time Object Detection Platform that proves you understand real outages, not happy-path diagrams.

Key Highlights

•Detection returns boxes + scores + class_id, not just top-1 label
•Video streams dominate QPS; single images are a minority path
•Post-process NMS is part of the serving critical path

1	@dataclass(frozen=True)
2	class DetectRequest:
3	tenant_id: str
4	frame_uri: str
5	model_id: str
6	score_threshold: float

1	export interface DetectRequest {
2	tenantId: string;
3	frameUri: string;
4	modelId: string;
5	scoreThreshold: number;
6	}