Design a Multi-Stream MUXer

Problem statement

Design a Multi-Stream MUXer that ingests multiple simultaneous video and audio sources, aligns them in time, composites them into one layout such as split-screen or picture-in-picture, optionally applies transitions and overlays, and publishes a single final stream for distribution. Examples include sports camera angles, esports observer feeds, live shopping hosts, remote interviews, church or stadium event production, and co-streamed creator shows.

This is not a standard upload or transcoding pipeline. The system must operate on live, lossy, jittery, independently clocked media. One source may arrive over SRT from a stadium encoder, another over WebRTC from a phone, another over RTMP from OBS, and another through a contribution link with intermittent loss. The muxer must keep lip sync stable, choose what appears on screen, tolerate missing or frozen feeds, and continue producing a broadcast-grade output even when an input degrades.

The architecture separates two planes. The control plane manages jobs, layouts, inputs, outputs, health, permissions, and operator intent. The media plane performs packet reception, clock recovery, jitter buffering, frame alignment, resampling, compositing, audio mixing, encoding, and packaging. The control plane can request a layout change, but the media plane must execute it frame-accurately and safely. A delayed API call must never cause a black output, an audio pop, or a half-rendered transition.

The hardest engineering problems are synchronization under imperfect clocks, deterministic low-latency compositing, GPU resource isolation, failover without visible disruption, and multi-tenant scaling. A strong answer treats the output stream as a safety-critical real-time artifact: missing data must be replaced by a designed fallback such as slate, freeze, muted audio, or backup camera, not by undefined behavior.