Design a Large-Scale Vector Database for Embeddings

Problem statement

Design a database whose primary key payload is a high-dimensional embedding (512D, 1024D, or 3072D floats produced by an LLM, a two-tower retriever, or a vision encoder) and whose primary query is top-k approximate nearest neighbor (ANN) search under a similarity metric, optionally constrained by metadata filters and fused with sparse lexical scores. The system must ingest vectors in real time as documents change, absorb periodic bulk re-embedding when the embedding model upgrades, scale to billions of vectors, and serve sub-second, usually sub-100ms, retrieval.

This is not a relational problem in disguise. In high dimensions there is no total order that prunes search: exact nearest neighbor degrades toward brute force, an O(N x d) scan per query, which is why every production system at billion scale commits to approximation. The design therefore owns four planes at once: an ingestion plane that turns documents into versioned vectors through a durable log; an index plane that builds and publishes immutable ANN segments; a query plane that scatters a query across shards, merges partial top-k heaps, filters, and reranks inside a latency budget; and a governance plane that measures recall as a deployed SLO, enforces tenant quotas, and makes deletion real.

Why the problem is distinctive

A key-value store retries a lost write. A vector database can lose a write, lose a replica, or load a stale segment and still answer, but the answer may be silently wrong: recall is a correctness property that degrades gradually instead of failing loudly. The second distinction is concurrency between mutable ingestion and immutable serving structures: HNSW-style graphs and PQ codebooks are expensive to build, so real systems separate a small mutable delta index that gives seconds-level visibility from sealed immutable segments that give millisecond serving, and swap them atomically through a versioned manifest. The third distinction is that filters change the algorithm, not just the result set: a predicate that matches 0.1 percent of vectors inverts which search strategy is correct.

Public baseline versus design assumptions

Public evidence shows the category is real at scale. Microsoft Research published DiskANN, reporting billion-point nearest neighbor search on a single node using an SSD-resident Vamana graph, and SPANN, reporting billion-scale search with a memory-plus-SSD posting hierarchy. Meta published embedding-based retrieval for Facebook Search over indexes of billions of social objects using FAISS-style sharded ANN leaves. Spotify open-sourced Annoy, memory-mapped random-projection tree forests, for recommendation-scale similarity. Pinecone and Zilliz publicly operate managed billion-vector indexes with serverless, object-storage-backed tiering. Those are cited company and paper claims, not our requirements.

For capacity planning this answer explicitly assumes a multi-tenant platform holding 12 billion vectors across 500 collections, a peak of 24,000 queries per second, and 1 percent daily churn. Unless tied to a citation, every number here is a stated design assumption, budget, or target.