Design Gesture Control System - System Design | WinJob

Problem Statement: Spatial Hand Input

Apple Vision, Google MediaPipe Hands, and Ultraleap Leap Motion all converge on the same interview story: turn unstable pixels into stable, low-latency digital commands without a controller.

Quantified anchors

| Metric | Value | Notes |

| Gesture frames | 30 fps | RGB-D or monochrome IR |

| End-to-end p95 | < 45 ms | pinch/select path |

Design decisions for this slice

Decision 1.1: landmark stream — 21-point hand mesh at 30 Hz from ToF/RGB

Decision 1.2: command debounce — hysteresis window before firing UI events

Decision 1.3: multi-user zones — spatial ROI per skeleton ID

Decision 1.4: on-device inference — CoreML/NNAPI path for privacy

Failure modes to pre-empt

Mirror inversion on front camera

Gesture-specific invariant (q-508-design-gesture-control)

Raw camera frames never leave the device unless the user opts into cloud training; only landmark tensors and gesture IDs cross the API.

Implementation sketch

javaOne Dark Pro

1 public final class LandmarkRingBuffer { private final float[][] slots = new float[64][63]; }

pythonOne Dark Pro

1 from dataclasses import dataclass
2  
3 @dataclass
4 class HandLandmarks:
5     points: list[tuple[float, float, float]]
6     timestamp_ms: int

typescriptOne Dark Pro

1 export function normalizeLandmarks(raw: Float32Array): Float32Array {
2   const palm = raw.slice(0, 3);
3   return raw.map((v, i) => v - palm[i % 3]);
4 }

Whiteboard checkpoint 1

Draw camera → preprocess → landmark → classify → debounce → OS/AR command bus.

Why interviewers care

Gesture Control System interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for Problem Statement: Spatial Hand Input that proves you understand real outages, not happy-path diagrams.

Key Highlights

•landmark stream
•command debounce
•multi-user zones
•on-device inference

1	from dataclasses import dataclass
2
3	@dataclass
4	class HandLandmarks:
5	points: list[tuple[float, float, float]]
6	timestamp_ms: int

1	export function normalizeLandmarks(raw: Float32Array): Float32Array {
2	const palm = raw.slice(0, 3);
3	return raw.map((v, i) => v - palm[i % 3]);
4	}