Design an Edge ML Inference Platform (Mobile/IoT)

Problem statement

Design an edge ML inference platform that builds, compresses, signs, distributes, activates, runs, and continuously improves machine-learning models on constrained devices: smartphones, cameras, gateways, and embedded boards. The platform must take a trained model from the model registry, run a compression and quantization pipeline, evaluate accuracy and latency on representative hardware, package signed per-target artifacts, roll them out in guarded waves, activate them atomically on each device through an A/B slot agent, collect privacy-preserving performance telemetry, detect drift or regression, and trigger re-quantization or partial retraining when warranted. If a device cannot run inference locally or its model is stale beyond policy, the platform falls back to a cloud inference service, but the fallback is the exception path, not the design center.

This is not a request/response model-serving API. The inference happens on hardware you do not control, behind cellular links you cannot guarantee, with batteries, thermal envelopes, and storage budgets you must respect. The cloud owns the pipeline, the registry, the rollout state machine, and the aggregate learning loop. The device owns the inference deadline. A delayed or lost packet must degrade a rollout or a dashboard, never a frame of real-time inference.

Why the problem is distinctive

A cloud model server can retry a request and scale replicas. An edge fleet cannot recall ten million deployed artifacts the way it recalls a container image, because devices are intermittently connected, heterogeneous, and physically in users' hands. The design therefore separates four concerns: artifact correctness (signed, versioned, evaluated before release), distribution efficiency (CDN waves, binary deltas, resumable downloads), activation safety (A/B slots, rollback, never brick), and learning integrity (aggregate, privacy-filtered signals that trigger the next pipeline run without raw data leaving the device).

Public operating baseline versus design assumptions

Public evidence shows this category is operationally real. Apple ships Core ML with dedicated Neural Engine hardware and states the A17 Pro delivers up to 35 TOPS on its Neural Engine; Google distributes TensorFlow Lite and ML Kit across Android and documents on-device federated learning in Gboard; NVIDIA's Jetson family publishes edge accelerators such as Jetson Orin Nano at 40 TOPS and manages fleets through its edge software stack; Qualcomm's AI Hub and Hexagon NPU publish tens of TOPS on mobile silicon. These are cited vendor figures and establish that on-device inference is a shipping product category, not research.

For capacity planning this answer explicitly assumes a mature platform: 12 million registered devices, 3.5 million daily active devices, 900 million on-device inferences per day, and rollout waves with a five-times reconnect peak. Unless tied to a citation, every number is a stated design assumption, budget, or target, not a company claim.

The four architectural planes

1. Model pipeline plane: training handoff, pruning, distillation, quantization (PTQ and QAT), per-hardware compilation, evaluation gates, packaging, signing.
2. Distribution plane: model registry, release channels, staged rollout waves, CDN delivery, delta patching, device agent download and verification.
3. Device runtime plane: model loading, accelerator delegation (NPU/GPU/DSP/CPU), memory arenas, latency governor, A/B slot activation, cloud-fallback decision.
4. Learning plane: privacy-preserving telemetry aggregation, latency and accuracy monitors, drift detection, feedback triggers into the pipeline.

A strong interview answer keeps these planes separate. The learning plane may propose a new model, but only the pipeline plane can produce a signed artifact, and only the distribution plane can activate it on a device.