Design a Multi-Party Audio Conference

Problem statement

Design a multi-party audio conference platform that lets participants join a shared room through a VoIP client or by dialing a PSTN phone number, hear every other active speaker with conversational quality, see who is talking, mute and unmute themselves, and be governed by a host. The system must mix or forward audio so that N participants can talk and listen simultaneously, handle overlap without clipping, keep echo below audibility, and survive media-node failures without dropping the room.

The core engineering fact is fan-out. In a naive peer-to-peer mesh, every participant sends their encoded audio to every other participant: with N participants each device uploads N-1 streams and downloads N-1 streams. At N=8, that is 7 upstream Opus streams of ~32 kbps plus RTP overhead, roughly 300 kbps up, which is already beyond what many mobile uplinks and hotel Wi-Fi networks sustain symmetrically. Mesh therefore collapses at small N, and everything interesting in this design exists to tame the N-to-N problem: a media plane that terminates every RTP stream once and replicates it, a mixing plane for endpoints that cannot decode many streams, a signaling plane that keeps the roster coherent, and a telephony plane that bridges the circuit-switched world into the same room.

Why audio-only is still hard

Audio-only removes video's bandwidth and codec complexity but raises the latency bar. Video viewers tolerate 500-800 ms of glass-to-glass delay; voice users notice overlap and talk over each other past roughly 250-300 ms one-way. ITU-T G.114 is the classic reference: one-way delay below 150 ms is acceptable for most user satisfaction, and above 400 ms becomes unacceptable for interactive conversation. That budget must cover capture, encoding, network queuing, jitter buffering, mixing, decoding, and playout, so every architectural hop is measured in tens of milliseconds. A design that routes media through a centralized REST backend for 'simplicity' fails before it starts.

Second, audio is unforgiving of loss in a different way than video. A dropped video frame degrades gracefully; a dropped 20 ms voice packet during a consonant creates a click or a swallowed syllable. The design needs forward error correction (FEC/RED), retransmission for non-real-time-critical repairs, packet loss concealment (PLC) in the decoder, and a jitter buffer that adapts to network conditions. All of this runs per participant, per direction, hundreds of times per second.

Third, PSTN interop changes the topology. A phone endpoint speaks G.711 over RTP inside a circuit-switched call, has no browser APIs, no data channel for roster events, and often no echo cancellation of its own. The platform needs a telephony gateway that terminates SIP trunks, transcodes between G.711 and Opus, injects the phone caller into the same conference fabric, and relays host controls through DTMF.

The four planes

1. Signaling plane: room creation, authentication, roster membership, host controls, join/leave events, and the WebSocket control path.
2. Media plane: RTP/UDP termination, SRTP decryption, selective forwarding (SFU) or mixing (MCU), congestion response, and per-conference routing.
3. Telephony plane: SIP trunk termination, PSTN dial-in/dial-out, transcoding, DTMF, and carrier interconnect.
4. Control and data plane: conference metadata, CDRs, recording manifests, analytics, and operational dashboards.

A strong answer keeps these planes separate so that the media plane can keep forwarding audio while the signaling plane is degraded, and so that a signaling outage degrades roster freshness without interrupting an ongoing conversation.

Public operating baseline versus design assumptions

Public figures establish the category's scale. Zoom reported 150 million daily meeting participants at its 2021 peak and roughly 100 billion daily meeting minutes in the same period. Discord has reported more than 200 million monthly active users, with voice a core surface. Agora's filings reported roughly 17.4 billion monthly engagement minutes across its real-time network. These are cited company figures, used as context. Every capacity number in this answer that is not attributed is an explicit design assumption, labeled as such.