Design Experiment Tracking - System Design | WinJob

Problem Statement: ML Experiment Tracking

Design an enterprise ML experiment tracking platform that logs runs, hyperparameters, time-series metrics, and artifacts with reproducible lineage. This section uses a mechanism-first lens on runs as first-class records with params, metrics, tags, and artifact URIs.

Why Weights & Biases, MLflow, and Neptune interviews probe here

Vendors need engineers who separate metadata OLTP from metric flood ingestion and who can defend compare-query latency without scanning raw time series. Hand-waving "we log to S3" fails when asked about idempotent batching, per-tenant noisy neighbors, or chart freshness under 18K writes/sec.

Operational detail you should voice aloud

State numeric assumptions: 85K runs/day, 4.2B metric points/day, 18K peak metric QPS, P99 log_metric 120 ms, and 190 TB/day artifact ingress. Tie each figure to a formula on the whiteboard.

Failure modes worth volunteering

Duplicate metric steps from retried SDK batches, chart staleness when rollup workers lag, artifact uploads without manifest closure, and compare queries that fan out to cold storage. For each, name detection (dedupe key collision metric, consumer lag alert) and mitigation (idempotency table, autoscaled rollup pods, upload completion webhook).

Whiteboard checkpoint

Draw API + Kafka + columnar metrics on the left, Postgres metadata + OpenSearch facets on the right, object store underneath. Label where idempotency keys land and where rollups feed the compare UI.

Implementation snippets (experiment tracking)

javaOne Dark Pro

1 public record RunKey(String workspaceId, String runId) {}
2 public enum RunStatus { RUNNING, FINISHED, FAILED, KILLED }

pythonOne Dark Pro

1 @dataclass(frozen=True)
2 class RunKey:
3     workspace_id: str
4     run_id: str
5  
6 class RunStatus(str, Enum):
7     RUNNING = "running"
8     FINISHED = "finished"
9     FAILED = "failed"
10     KILLED = "killed"

typescriptOne Dark Pro

1 export interface RunKey {
2   workspaceId: string;
3   runId: string;
4 }
5 export type RunStatus = "running" | "finished" | "failed" | "killed";

Section-specific depth (sec-01)

Anchor this slice to Problem Statement: ML Experiment Tracking: explain how workspace → project → experiment → run hierarchy changes chart lag SLIs, how time-series metrics separate from oltp metadata affects retention cost, and how reproducibility bundle: git sha, docker image, seed, data snapshot uri shapes SDK retry contracts—not generic MLOps platitudes.

Why interviewers care

Experiment Tracking interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for Problem Statement: ML Experiment Tracking that proves you understand real outages, not happy-path diagrams.

Key Highlights

•Workspace → Project → Experiment → Run hierarchy
•Time-series metrics separate from OLTP metadata
•Reproducibility bundle: git SHA, docker image, seed, data snapshot URI
•Compare runs UI is read-heavy aggregation, not raw log replay

1	public record RunKey(String workspaceId, String runId) {}
2	public enum RunStatus { RUNNING, FINISHED, FAILED, KILLED }

1	@dataclass(frozen=True)
2	class RunKey:
3	workspace_id: str
4	run_id: str
5
6	class RunStatus(str, Enum):
7	RUNNING = "running"
8	FINISHED = "finished"
9	FAILED = "failed"
10	KILLED = "killed"

1	export interface RunKey {
2	workspaceId: string;
3	runId: string;
4	}
5	export type RunStatus = "running" \| "finished" \| "failed" \| "killed";