Design a VR/AR Streaming Service

Problem statement

Design a VR/AR streaming service that delivers immersive experiences over the network. Two delivery families must coexist. First, cloud-rendered pixel streaming: a GPU server renders the world, encodes each eye's frame, ships it over a lossy network, and the headset displays it while sending head, eye, and hand pose back at high frequency. Second, viewport-adaptive streaming of pre-baked immersive content: tiled 360-degree video or volumetric scenes where the client renders or composites locally and only the pixels inside the user's predicted viewport are delivered at full quality.

This is not Netflix with a wider aspect ratio. In 2D streaming the viewer is passive and a 300 ms buffer is invisible. In VR the viewer's vestibular system is inside the loop: the displayed image must track head rotation within tens of milliseconds or the user feels nausea. The platform is therefore a closed control loop — pose out, pixels back — and every layer is judged by its contribution to motion-to-photon latency, not just throughput or uptime.

Why the problem is distinctive

Three properties separate this from ordinary media systems. (1) The client controls the server's workload in real time. Every head turn invalidates previously rendered pixels, so caching in the traditional CDN sense barely applies to interactive sessions. (2) Bandwidth is extreme and asymmetric. A single 90 Hz stereoscopic AV1 session consumes 60-150 Mbps sustained, while the return path carries millions of tiny pose datagrams per second across the fleet. (3) Failure has physiological consequences. Frame-rate collapse, latency spikes, or stutter produce motion sickness and can break the product's trust permanently.

The two planes of the architecture

The design splits cleanly into an experience plane and a media plane. The experience plane owns sessions, entitlements, billing, content catalog, world anchors for AR, social presence, and analytics. It is eventually consistent, horizontally scalable, and tolerant of hundreds of milliseconds. The media plane owns the pose-to-pixel loop: GPU placement, render queues, encoding, transport, congestion control, prediction, and reprojection. It is latency-critical, regional, and must degrade gracefully rather than buffer.

Public operating baseline versus design assumptions

The category is real and shipped. NVIDIA CloudXR streams VR and AR from RTX-class GPU servers to headsets using NVENC hardware encoding and is used for enterprise training. Virtual Desktop ships PC VR to Quest headsets over Wi-Fi using H.264, HEVC, and 10-bit AV1 at up to roughly 200 Mbps. Meta launched cloud streaming on Quest 3 in 2025 and has published research on foveated encoding claiming around 50 percent bitrate reduction with eye tracking. These company-reported facts set context; every fleet size, SLO, and capacity number used below for our fictional service is an explicitly stated design assumption.

For capacity planning this answer assumes a mature service with 1.2M daily active users, 350K concurrent sessions at global peak, 120K of which are cloud-rendered interactive sessions, an average bitrate of 72 Mbps after foveation, and a five-times signaling peak during launches and live 360 events.