Design Live Captions

Scope and Product Boundaries

Live captions are a real-time accessibility feature, not a batch transcript feature with a faster clock. The product must turn active meeting audio into readable text while people are still deciding whether to interrupt, ask for clarification, or follow a speaker they cannot hear clearly.

I would scope v1 to meeting and webinar captions for authenticated participants. The system receives mixed or per-participant audio, emits mutable partial hypotheses quickly, upgrades them into final caption segments, and optionally exports a post-call transcript if retention policy allows it. Translation, summarization, and searchable meeting intelligence are adjacent products; they should consume final segments later rather than slow the caption hot path.

The core invariant is that the viewer never sees text move backward in time. A partial hypothesis can be revised inside a short correction window, but finalized spans become immutable with stable start/end offsets, speaker attribution, language metadata, and policy labels. This keeps the live UX responsive while preserving a defensible transcript record.

A senior design answer also names the policy boundary early. Profanity masking, PII redaction, recording consent, speaker labels, and transcript retention vary by tenant, region, and meeting type. Those policies should be evaluated before export and during finalization, while the emergency fallback remains a minimal readable caption stream.

The interview stance is simple: optimize first for human comprehension under 1.5 seconds p95 partial latency, then use asynchronous rescoring to improve punctuation, casing, diarization, and domain terminology without disrupting the words already shown to users.

Why interviewers care

Live Captions interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

The failure that defines the design

The outage to narrate is caption lag. Under load the ASR workers or their GPUs saturate, the audio queue backs up, and captions start appearing five or ten seconds behind the speaker — for a deaf participant relying on them to follow a live conversation, captions that lag the speech are worse than useless. The fix is structural and is the spine of the whole design: stream the recognizer so it emits low-confidence partial captions within a fraction of a second and rewrites them as it hears more, finalize on a natural pause, and shed to a faster model or coarser updates under load rather than letting the queue grow. State the partial-then-final streaming model up front, because live captions are a latency problem first and an accuracy problem second.