Design an Auto Tagging / Face Recognition Feature

Problem statement

Design an auto-tagging / face-recognition feature for a photo-sharing social network. When a user uploads a photo, the system must detect every human face in the image, generate a compact embedding for each, compare those embeddings against a gallery of known users, and suggest tags — "This looks like Alice, Bob, and Carol" — either for the uploader to confirm or, if the user has opted in, applied automatically.

This is not a toy computer-vision exercise. The feature sits inside a production social platform that already handles uploads, feeds, notifications, privacy settings, and content moderation. The face-recognition pipeline must integrate with all of those subsystems without becoming a bottleneck or a privacy liability.

Why the problem is distinctive

A search engine can tolerate a wrong result; the user simply does not click it. A face-recognition tag is different. A false-positive tag publicly associates the wrong person with a photo, creating social harm, legal exposure under biometric-data laws (GDPR Article 9, Illinois BIPA, CCPA), and trust damage that compounds virally. A false-negative tag is merely a missed convenience. The asymmetry between false-positive cost and false-negative cost shapes every threshold, every fallback, and every user-consent flow in the design.

The four architectural planes

1. Ingestion plane: photo upload, EXIF extraction, content-moderation check, image normalization, and event emission into the processing pipeline.
2. Inference plane: face detection, alignment, embedding generation, and similarity search against the known-face gallery.
3. Tagging plane: suggestion ranking, user-confirmation workflow, auto-tag policy engine, and social-graph enrichment.
4. Governance plane: consent management, opt-out propagation, biometric-data retention, audit logging, and regulatory compliance.

A strong interview answer keeps these planes separate. The inference plane can degrade or queue without blocking uploads. The governance plane can revoke embeddings without touching the photo. The tagging plane can fall back to manual tagging without losing any photos.

Public operating baseline

Meta launched face-recognition tagging in 2010 and by 2017 was processing it on over a billion photos per day across Facebook and Instagram. The DeepFace model published in 2014 achieved 97.35% accuracy on the LFW benchmark. Google Photos, launched in 2015, used FaceNet embeddings and on-device clustering. Apple Photos performs all face recognition on-device via the Neural Engine, never uploading embeddings. Amazon Rekognition and Azure Face API offer managed alternatives. Each represents a distinct architectural trade-off between accuracy, latency, cost, and privacy.

For capacity planning, this answer explicitly assumes a mature social platform with 200 million daily active users, 350 million photo uploads per day, an average of 2.3 faces per photo, and a known-face gallery of 800 million unique face embeddings. Unless a number is tied to a citation, it is a stated design assumption.