Design Embedding Service - System Design | WinJob

Problem Statement: Enterprise Embedding Platform

Design a central embedding platform that turns text, code snippets, and optional images into dense vectors for downstream search, RAG, and ranking. Unlike a one-off model endpoint, this is multi-tenant infrastructure with quotas, versioning, and cost controls—similar in spirit to OpenAI Embeddings, Cohere Embed, or Google Vertex text-embedding models.

Core user journeys

Product engineer calls POST /v1/embeddings from a search indexer with 2–8 KB documents, expecting p99 < 120 ms for short text on a warm GPU pool.

Data platform submits a batch job over 50M rows in Parquet, receiving vectors in object storage with a manifest keyed by content_hash and model_version.

ML platform registers text-embedding-3-large@2026-05-01, runs offline eval (MTEB slice), then promotes 5% canary before full cutover.

Why this is a hard interview

Embeddings look simple (forward pass) but production systems must solve deduplication, dimension contracts, GPU bin-packing, rate fairness, and backward-compatible model upgrades without reindexing the entire corpus overnight.

Real-world anchors

OpenAI reports billions of embedding calls/month; Cohere optimizes batch APIs for bulk ingestion; Google publishes embedding endpoints with regional routing. Your design should cite these patterns while staying implementation-agnostic.

javaOne Dark Pro

1 public record EmbedRequest(String tenantId, String model, List<String> inputs, String inputType) {}

pythonOne Dark Pro

1 @dataclass(frozen=True)
2 class EmbedRequest:
3     tenant_id: str
4     model: str
5     inputs: list[str]
6     input_type: str  # query | document

typescriptOne Dark Pro

1 export interface EmbedRequest {
2   tenantId: string;
3   model: string;
4   inputs: string[];
5   inputType: "query" | "document";
6 }

Interview signal

Frame the embedding service as multi-tenant infrastructure, not a model endpoint: content-hash dedup, immutable dimension contracts, GPU bin-packing, per-tenant fairness, and backward-compatible model upgrades without reindexing the whole corpus. The tell is naming a real failure — a model bump that silently changes vector geometry — and holding the p99 < 120 ms warm-pool target.

Key Highlights

•A central embedding platform turns text, code, and images into dense vectors for search, RAG, and ranking.
•Multi-tenant infrastructure: quotas, immutable versioning, and cost controls — not a one-off endpoint.
•Hard parts: content-hash dedup, dimension contracts, GPU bin-packing, fairness, and zero-reindex upgrades.
•Warm-pool target p99 < 120 ms for short text; the batch path writes vectors plus a manifest to object storage.

1	@dataclass(frozen=True)
2	class EmbedRequest:
3	tenant_id: str
4	model: str
5	inputs: list[str]
6	input_type: str # query \| document