Design a Dedicated Feature Store for ML

Problem statement

Design a dedicated feature store for machine learning: a repository where ML features are consistently defined, computed, stored, updated, and served to both training pipelines (batch, terabyte scale, historically correct) and real-time inference (online, millisecond lookups of the latest values). The system must guarantee consistent feature definitions and values across both planes, support batch ingestion for historical features, real-time updates for streaming features, versioning and schema management, and access control for many model teams.

The hard part is not storage. The hard part is that training and serving have opposite access patterns and opposite failure semantics. Training scans billions of historical rows and requires point-in-time correctness: for each labeled example at time T, the features must reflect exactly what was knowable at T, with no leakage from the future. Inference reads a handful of entity keys and requires the freshest value within a few milliseconds. If the two planes compute features with different code, different timing, or different handling of nulls and late data, models silently degrade in production. This divergence is called training-serving skew, and eliminating it is the defining requirement of a feature store.

Why this problem exists

Without a feature store, every model team writes its own SQL for training and its own service code for serving. The same 'user_7d_purchase_count' gets implemented three times by three teams, each with slightly different window boundaries and timezone handling. Reuse is impossible because nobody can discover or trust someone else's feature. Backfills are bespoke. Auditors cannot reconstruct which feature values a shipped model actually saw. The feature store centralizes definition, computation, materialization, serving, and monitoring so that a feature is written once, governed once, and reused everywhere.

The four architectural planes

1. Registry plane: the source of truth for feature definitions, schemas, versions, ownership, data sources, and access policy. Everything else is derived from it.
2. Offline plane: batch transformation and materialization of historical feature values into a columnar store, plus point-in-time-correct training dataset assembly and backfills.
3. Online plane: streaming ingestion and incremental updates into a low-latency store holding the latest feature values per entity key.
4. Serving and observability plane: low-latency retrieval APIs and SDKs for inference, feature logging, drift and skew monitoring, and lineage.

A strong answer keeps these planes separate and then shows the contracts that bind them: one definition in the registry, one materialization contract between offline and online, one retrieval contract for serving.

Public evidence that this category is real

Uber publicly described Michelangelo in 2017, including a feature store that computes features offline and serves them from an online store; Uber engineering posts describe Cassandra-backed online feature serving for prediction services. Feast originated at Gojek around 2018 to give fraud and dispatch models consistent features, and was open-sourced in 2019; Robinhood has publicly described using Feast for ML feature serving. LinkedIn open-sourced Feathr in 2022 and described it as powering hundreds of production ML use cases at LinkedIn. Managed offerings followed: AWS SageMaker Feature Store became generally available in December 2020, Google Vertex AI Feature Store reached preview in 2021, and Azure Machine Learning feature store reached general availability in 2023. These anchors show the interviewer you are designing a known, proven category, not inventing one.

Design baseline

Unless tied to a public source, every number in this answer is an explicit design assumption. The baseline: a mature enterprise ML platform serving about 50 model teams and 500 data scientists and ML engineers, with 3,000 feature views, 30,000 registered features across 20 entity types, an offline store of roughly 500 TB of materialized features, streaming ingestion around 500,000 events per second, and online serving at 50,000 requests per second average with a 150,000 request peak and a p99 budget of 10 milliseconds.