Problem Statement: A Latency-Critical Perception Loop, Not a Cloud API
Frames gesture control as an edge-first perception system whose photon-to-command loop must stay local, with the cloud owning sessions, training, and model delivery.
Problem statement
Design a Gesture Recognition & Control System that reads a camera or depth-sensor stream from an AR/VR headset or room rig, extracts hand or body keypoints, classifies gestures such as pinch, swipe, wave, grab, and point, maps them to application commands, and delivers those commands into a live AR/VR environment or game. It must survive partial occlusion, support more than one user gesturing at once, keep latency low enough that the interaction feels instantaneous, and optionally let users train their own gestures.
The single most important framing decision is this: the perception-to-command loop must run on-device. A user who pinches to select a hologram expects the selection to register within a few tens of milliseconds. If every frame round-trips to a cloud model, ordinary network latency and jitter make the interaction feel sluggish and, in VR, can contribute to discomfort. Therefore the capture → keypoint → classification → command path executes locally on the XR device or an edge node co-located with the room. The cloud owns everything that is not on the latency-critical path: multi-user session coordination, custom-gesture training, model packaging and over-the-air delivery, telemetry, analytics, and policy.
Why this problem is distinctive
A batch image-classification service can retry a prediction. A gesture system cannot retry a moment: the user has already moved on. The design therefore separates interaction correctness (did the right command fire at the right time for the right user?) from session and learning workflows (durable, eventually consistent, retryable). The interaction loop is a continuously evaluated, local, latency-bounded process. Session and learning are durable cloud workflows. Keeping these two planes separate is what distinguishes a credible AR/VR input system from a generic computer-vision backend.
The problem requires sensor or camera feed with keypoint detection, ML-based gesture classification, mapping gestures to commands, multi-user or multi-limb concurrency, real-time feedback for immersion, scalability across cameras and users, reliability under partial occlusion, and possibly a UI for training custom gestures.
Public operating baseline versus design assumptions
Public evidence shows this category is real and large. Meta has shipped hand tracking on the Quest line using four grayscale inside-out cameras and on-device models. Microsoft Kinect sold tens of millions of units and tracked a 25-joint skeleton at 30 frames per second, later evolving into HoloLens 2 articulated hand tracking. Google's MediaPipe ships open-source hand models that run in real time on phones. Ultraleap sells dedicated stereo-infrared hand-tracking hardware used in automotive and retail. These are cited company signals, not requirements for our fictional system.
For capacity planning, this answer explicitly assumes a mature platform with 8 million registered devices, 1.5 million daily active users, 3.6 million sessions per day, and 350,000 concurrent sessions at peak, with a five-times event-peak provision. Unless a number is tied to a citation, it is a stated design assumption, target, budget, or illustrative threshold—not a claim about any company's private architecture.
The four architectural planes
- Perception plane: sensor capture, time synchronization, hand/body keypoint detection, temporal tracking and smoothing. On-device.
- Interpretation plane: gesture classification, confidence gating, debounce and hysteresis, command mapping. On-device.
- Session plane: room and session state, multi-user arbitration, command dispatch to the application, developer SDK. Cloud plus edge.
- Learning plane: telemetry, custom-gesture recording and training, model evaluation, signed OTA model delivery. Cloud.
A strong interview answer keeps these planes separate. It lets the session plane degrade without breaking the local interaction loop, and it lets the learning plane improve models without silently changing the validated on-device recognition envelope.
Key Highlights
- •The photon-to-command loop runs on-device; the cloud never sits inside the gesture-to-feedback critical path.
- •Separate interaction correctness (local, latency-bounded) from session and learning workflows (durable, eventually consistent).
- •Public figures from Meta, Microsoft, Google, and Ultraleap provide context; every uncited SLO here is an explicit design assumption.
- •The architecture has four planes: perception, interpretation, session, and learning.
- •An interaction that never fires is better than a wrong command that fires randomly.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate the latency-critical interaction loop from the durable session and learning workflows before choosing services."
- "Let me first define which decisions are allowed on-device and which belong in the cloud."