Design an Audio Fingerprinting Service

Problem statement

Design an audio fingerprinting service that creates a compact, robust signature from raw audio and matches it against a large catalog. When new audio is uploaded to a platform, or when a user records a few seconds of music from their microphone, the system must decide whether that audio corresponds to a known track, a fragment of one, or a duplicate or derivative reupload. It must then feed the result into a rights workflow: identifying duplicates, flagging copyrighted material, and supporting takedown, monetization, or tracking outcomes.

The defining property of this problem is that matching happens on the signal itself, not on metadata. A reuploader strips ID3 tags, changes the container from FLAC to MP3, re-encodes at a lower bitrate, trims two seconds of silence, lowers the volume, or records a live playback with a phone microphone in a noisy cafe. ISRC codes and filenames tell you nothing. The fingerprint must be invariant to exactly these transformations: re-encoding, resampling, mono downmix, normalization, trimming, time offset, and moderate additive noise. At the same time it must remain discriminative enough that two different recordings almost never collide.

Two workloads, one index

This answer designs both product families that share the same core index:

1. Recognition queries (Shazam-like): short, noisy fragments of 5-20 seconds arrive from devices and must return a match in near real time. Apple announced in 2024 that Shazam had passed 2 billion total song identifications, with more than 200 million monthly average users, which anchors the scale of this workload.

1. Ingest and upload scanning (Content-ID-like): every new asset entering a catalog or UGC platform is fingerprinted and matched against the reference catalog to detect duplicates and unauthorized reuploads. YouTube reported in 2019 that more than 500 hours of video are uploaded per minute, and its Content ID pipeline scans that stream against rights-holder reference files at ingest. Spotify has publicly confirmed that it receives well over 100,000 new track uploads per day, each of which benefits from duplicate and rights screening.

Why metadata systems fail here

A naive design treats this as a lookup problem: hash the file bytes, store the hash, compare on upload. Byte hashing catches only bit-identical copies, which is a tiny fraction of real infringement and duplicate traffic. Re-encoding changes every byte while the perceptual content is unchanged. Fingerprinting solves the perceptual equivalence question: same recording, different encoding. A harder adjacent question, similar but not identical recording (covers, remixes, live versions, sped-up edits), needs additional mechanisms such as pitch-invariant features or learned embeddings, which this design includes as a second-stage module rather than pretending the hash solves it.

The four architectural planes

1. Ingestion and extraction plane: decode, normalize, compute spectral features, emit fingerprint hashes, manage asset versions.
2. Index plane: an inverted hash index mapping fingerprint hashes to (track, time offset) postings, sharded and tiered across memory and SSD.
3. Match and decision plane: candidate retrieval, time-offset alignment, scoring, thresholding, and verification.
4. Rights and action plane: claim creation, policy evaluation (block, monetize, track), dispute handling, and audit.

A strong answer keeps these planes separate. Extraction can lag without breaking recognition availability. The index can be rebuilt or resharded without touching claims. The rights plane can freeze automated actions during an incident without stopping matching. This separation is the backbone of the rest of the design.