Design Model Serving - System Design | WinJob

Problem Statement: Enterprise Model Serving Platform

Design a production model serving platform that registers trained models, routes online and batch inference traffic, and operates GPU fleets with safe rollouts.

Why interviewers probe here

Google, Amazon, and Meta run thousands of models in production; interviews test whether you treat inference as a distributed systems problem—not a notebook export.

Operational detail

State the serving contract up front: models are immutable artifacts addressed by name+version; traffic is routed by model ID and optional traffic split for canaries.

Failure and edge cases

Cold starts after scale-to-zero, OOM on oversized batches, version skew when router cache lags registry, and thundering herd when a viral model spikes.

Interview checkpoints

Separate training pipeline from serving hot path
Name the three personas: ML engineer, platform SRE, product client

javaOne Dark Pro

1 public record ModelRef(String name, String version, String stage) {}

pythonOne Dark Pro

1 @dataclass(frozen=True)
2 class ModelRef:
3     name: str
4     version: str
5     stage: str  # prod|canary|shadow

typescriptOne Dark Pro

1 export interface ModelRef { name: string; version: string; stage: "prod" | "canary" | "shadow"; }

Interview signal

Open by stating the serving contract — immutable name+version artifacts, routed by model ID with an optional canary split — and name a real failure (version skew when the router cache lags the registry). The tell is treating inference as a distributed-systems problem with SLOs, not a notebook export.

Key Highlights

•Control plane: registry, versioning, rollout policies
•Data plane: GPU workers with dynamic batching and autoscaling
•Clients: REST/gRPC predict, batch scoring, streaming embeddings
•Observability: latency SLOs, drift, GPU saturation

1	@dataclass(frozen=True)
2	class ModelRef:
3	name: str
4	version: str
5	stage: str # prod\|canary\|shadow