Design ML Pipeline Orchestration - System Design | WinJob

Problem Statement: ML Pipeline Orchestration

Design a multi-tenant ML pipeline orchestration platform that schedules DAGs of training, featurization, evaluation, and deployment steps across Kubernetes GPU pools—comparable to Kubeflow Pipelines, Apache Airflow with ML operators, or Google Vertex AI Pipelines. This section applies a scope and orchestration vocabulary lens on DAG runs; task instances; artifact lineage; multi-tenant quotas.

Why Kubeflow, Airflow, and Vertex interviews probe orchestration

Interviewers want proof you understand control-plane durability separate from ephemeral workers. A pipeline run is a state machine: parse DAG → enqueue root tasks → respect data dependencies → retry with backoff → checkpoint artifacts → emit lineage. Saying "we use CronJobs" fails when asked about backfill, dynamic task mapping, spot preemption, or cross-DAG sensors.

Operational numbers to voice on the whiteboard

Anchor sizing with explicit assumptions: 4,200 pipeline runs/day, 68,000 task instances/day, 480 concurrent runs, 920 metadata read/write QPS, and 38 TB/day of versioned artifacts. Tie each figure to a formula (runs × tasks/run × payload).

Failure modes to volunteer proactively

Scheduler leader loss during backfill, poison sensors blocking DAG parse, GPU node preemption mid-training without checkpoint, metadata DB hot rows on dag_run status updates, and artifact GC deleting tensors still referenced by downstream runs. For each, name the detector (stuck task SLA, lineage orphan scan) and mitigation (leader election, task timeout, resumable checkpoints, sharded status table, reference-counted blob store).

Whiteboard checkpoint (understanding)

Sketch metadata service (Postgres + outbox), queue (Kafka/Pub/Sub), executor (K8s/Argo/Batch), and artifact URI scheme (s3://tenant/run/step/output). Label idempotency keys on task attempts.

Section-specific depth

Contrast batch ETL orchestration with GPU-bound training graphs where tasks may run for hours and checkpoints dominate recovery.

Implementation anchors

javaOne Dark Pro

1 public record RunKey(String tenant, String pipelineId, String runId) {}
2 public enum TaskState { QUEUED, RUNNING, SUCCESS, FAILED, UPSTREAM_FAILED, SKIPPED }

pythonOne Dark Pro

1 @dataclass(frozen=True)
2 class RunKey:
3     tenant: str
4     pipeline_id: str
5     run_id: str
6  
7 class TaskState(str, Enum):
8     QUEUED = "queued"
9     RUNNING = "running"
10     SUCCESS = "success"
11     FAILED = "failed"

typescriptOne Dark Pro

1 export interface RunKey {
2   tenant: string;
3   pipelineId: string;
4   runId: string;
5 }
6 export type TaskState =
7   | "queued"
8   | "running"
9   | "success"
10   | "failed"
11   | "upstream_failed";

Why interviewers care

ML Pipeline Orchestration interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for Problem Statement: ML Pipeline Orchestration that proves you understand real outages, not happy-path diagrams.

Key Highlights

•DAG runs
•task instances
•artifact lineage
•multi-tenant quotas

1	public record RunKey(String tenant, String pipelineId, String runId) {}
2	public enum TaskState { QUEUED, RUNNING, SUCCESS, FAILED, UPSTREAM_FAILED, SKIPPED }

1	@dataclass(frozen=True)
2	class RunKey:
3	tenant: str
4	pipeline_id: str
5	run_id: str
6
7	class TaskState(str, Enum):
8	QUEUED = "queued"
9	RUNNING = "running"
10	SUCCESS = "success"
11	FAILED = "failed"

1	export interface RunKey {
2	tenant: string;
3	pipelineId: string;
4	runId: string;
5	}
6	export type TaskState =
7	\| "queued"
8	\| "running"
9	\| "success"
10	\| "failed"
11	\| "upstream_failed";