Design Multimodal AI System - System Design | WinJob

Problem Statement: Production Multimodal AI Platform

Design a multimodal AI platform that accepts text, images, audio, and short video in one session, fuses them into a shared representation, and drives downstream tasks: grounded Q&A, captioning, visual reasoning, and tool use. This is the architecture class behind GPT-4o/Gemini native multimodal APIs—not a demo that resizes images and stuffs them into a text prompt.

Primary journeys

Product analyst uploads a dashboard screenshot plus a CSV export and asks "Which region drove the Q3 dip?" The system OCRs the chart, aligns series labels with tabular columns, and answers with citations to both modalities within p95 6s for a 2-image + 1k-token text turn.

Field technician streams 10s of machine audio while pointing the camera at a panel; the platform transcribes phonemes, detects alarm tones, and runs a vision classifier on LED states—returning a ranked fault hypothesis with confidence and escalation to human review below 0.7 fused score.

Safety reviewer audits why the model refused a borderline medical image: every inference stores modality hashes, encoder versions, policy rule ids, and redacted prompt spans for GDPR Article 17 deletes within 24h.

Why interviews probe this hard

Candidates collapse multimodal into "call CLIP then GPT." Staff answers separate modality ingest & normalization, alignment / fusion encoders, router & GPU pools per modality, context assembly with token economics, guardrails per modality, and offline eval (hallucination on charts, OCR drift, ASR word-error rate).

Scale anchors

Assume 8k enterprise tenants, 3.5M multimodal sessions/day, average 2.4 modalities per request, ~18k visual tokens equivalent per heavy turn after tiling, and 120M media assets in object storage with per-tenant encryption keys.

javaOne Dark Pro

1 // Illustrative contract — production adds retries, tracing, and policy hooks
2 public record ModalityEnvelope(String tenantId, String sessionId, List<ModalityPart> parts) {}

pythonOne Dark Pro

1 from dataclasses import dataclass
2  
3 @dataclass(frozen=True)
4 class ModalityEnvelope:
5     tenant_id: str
6     session_id: str
7     parts: list  # image | audio | text | video refs

typescriptOne Dark Pro

1 export interface ModalityEnvelope {
2   tenantId: string;
3   sessionId: string;
4   parts: ModalityPart[];
5 }

Why interviewers care

Multimodal AI System interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for Problem Statement: Production Multimodal AI Platform that proves you understand real outages, not happy-path diagrams.

1	// Illustrative contract — production adds retries, tracing, and policy hooks
2	public record ModalityEnvelope(String tenantId, String sessionId, List<ModalityPart> parts) {}

1	from dataclasses import dataclass
2
3	@dataclass(frozen=True)
4	class ModalityEnvelope:
5	tenant_id: str
6	session_id: str
7	parts: list # image \| audio \| text \| video refs