Design a Data Labeling Platform with Active Learning

Problem statement

Design a data labeling platform where human labelers annotate images, text, and audio, and the platform's own model decides which unlabeled items are most valuable to label next. The model scores every candidate asset for uncertainty, the selection engine converts scores into prioritized task queues, labelers consume those queues, submitted labels flow back into a versioned training set, a retraining job refreshes the model after each batch, and the refreshed model re-ranks the remaining pool. This is a closed loop: every human minute changes what the machine asks for next, and every machine decision changes what humans see.

The problem requires a labeling UI for images, text, or audio; model-based sample selection via uncertainty and margin sampling; logging and metrics for labeling speed and accuracy; and possible retraining after each batch, all while handling partial annotation reviews and many concurrent labelers. The competitive gap named in the brief is concurrency around active-learning queries: most shallow answers draw a labeling app and stop. The hard part is bridging a slow, expensive, fallible human workforce with a fast, cheap, probabilistic model that must keep learning without poisoning itself.

Why this problem is distinctive

A generic annotation tool can treat tasks as static rows: create, assign, submit, done. An active-learning platform cannot, for four reasons. First, the queue is alive: task priority changes every time the model retrain completes, so assignment must tolerate re-ranking without starving labelers or double-issuing tasks. Second, the label is a random variable: humans disagree, so the platform must collect redundancy, measure agreement, and merge conflicting submissions into one trusted consensus label. Third, the model is both customer and product: it consumes labels and produces selection scores, so a bad batch of labels degrades the model, which degrades selection, which wastes more human minutes — a feedback loop that can amplify errors. Fourth, the scarce resource is human attention, not compute: the architecture must maximize the information gained per label, which is exactly what active learning is for.

Published research sets the expectation. Lewis and Gale showed in 1994 that uncertainty sampling could cut the labels needed for text classification by more than an order of magnitude compared with passive sampling. Amazon has described SageMaker Ground Truth as using active learning to auto-label high-confidence examples while routing uncertain ones to humans, reporting that automated labeling can substantially reduce per-label cost at scale. So the interview answer must quantify label savings, not just draw boxes.

The four architectural planes

1. Workforce plane: labelers, reviewers, admins, skills, payments, capacity.
2. Workflow plane: task lifecycle, claims, drafts, submissions, consensus, review.
3. Learning plane: model cohorts, uncertainty scoring, batch selection, retraining triggers.
4. Data and governance plane: asset store, dataset versions, provenance, privacy, compliance.

A strong answer keeps these planes separate: the workforce plane may degrade without corrupting labels, the learning plane may lag without blocking submissions, and governance constraints apply to every plane at once.