Design Feature Store - System Design | WinJob

Feature Store Problem Framing

A feature store is the contract between the data that produces features and the models that consume them: it guarantees that the value a model trained on is the value it serves on. The whole job is to kill training-serving skew — the silent failure where offline accuracy looks great but production regresses because the online feature was computed differently or arrived stale.

Scale to design for: 1.8M active entities/day, 11B stream events/day, 3.1M peak online batchGet QPS at p99 < 12 ms, 4200 registered feature columns, and train/serve parity < 0.03%. Reason from those numbers, not from a box diagram.

The core invariant: every feature value is keyed by (entity_id, event_timestamp, definition_version). That is what lets yesterday's champion reproduce its exact training rows, and lets today's serving path attach freshness and version metadata to every vector instead of hard-failing on a stale read.

Quantified controls

Control	Target	Rationale
Online read p99	< 12 ms	Keeps two-tower ranking inside a 40 ms total budget
Parity sample	0.1% entities/hour	Catches skew before model promotion
Materialization lag	p95 < 90 s	Protects fraud features during traffic spikes
Train/serve parity	< 0.03%	Promotion gate; blocks skewed models
Registry availability	99.97%	Blocks unsafe feature promotions

Mechanism

One transformation artifact, registered in the control plane, feeds both the batch and stream materializers. The moment a feature is re-implemented as separate Spark and Flink SQL, skew creeps back in. Every write preserves (entity_id, event_timestamp, definition_version), which is exactly what makes a point-in-time training read and an online serving read return the same number.

Failure modes specific to feature stores

A model promoted without a parity-report id, so skew surfaces in production instead of at the gate.
Backfill jobs writing future-timestamped rows into offline partitions, leaking labels into training.
The same embedding registered under two column names, so trainers and servers silently diverge.
Sidecar caches still serving definition v3 while trainers read v2 snapshots.

Implementation slice

javaOne Dark Pro

1 public final class FeatureRecord {
2   private final String entityId;
3   private final long eventTimeMs;
4   private final int version;
5   private final Map<String, Double> values;
6  
7   public boolean isValidForLabel(long labelTimeMs) {
8     return eventTimeMs <= labelTimeMs;
9   }
10 }

pythonOne Dark Pro

1 @dataclass
2 class FeatureRecord:
3     entity_id: str
4     event_time_ms: int
5     version: int
6     values: dict[str, float]
7  
8 def valid_for_label(record: FeatureRecord, label_time_ms: int) -> bool:
9     return record.event_time_ms <= label_time_ms

typescriptOne Dark Pro

1 interface FeatureRecord {
2   entityId: string;
3   eventTimeMs: number;
4   version: number;
5   values: Record<string, number>;
6 }
7  
8 export function validForLabel(record: FeatureRecord, labelTimeMs: number): boolean {
9   return record.eventTimeMs <= labelTimeMs;
10 }

Interview signal

Frame the feature store as a skew-prevention contract, then name the one SLI you would dashboard in week one — parity mismatch rate — and explain how it gates promotion. The strong-candidate tell is talking in parity, freshness, and version numbers, not in box names.

Key Highlights

•A feature store guarantees the value a model trained on equals the value it serves — killing train/serve skew.
•Design from real numbers: 1.8M entities/day, 11B events/day, 3.1M peak QPS at p99 < 12 ms, parity < 0.03%.
•Every value is keyed by (entity_id, event_timestamp, definition_version) for reproducibility and PIT correctness.
•One shared transformation artifact feeds batch and stream — re-implementing it as separate SQL is how skew returns.

1	public final class FeatureRecord {
2	private final String entityId;
3	private final long eventTimeMs;
4	private final int version;
5	private final Map<String, Double> values;
6
7	public boolean isValidForLabel(long labelTimeMs) {
8	return eventTimeMs <= labelTimeMs;
9	}
10	}

1	@dataclass
2	class FeatureRecord:
3	entity_id: str
4	event_time_ms: int
5	version: int
6	values: dict[str, float]
7
8	def valid_for_label(record: FeatureRecord, label_time_ms: int) -> bool:
9	return record.event_time_ms <= label_time_ms

1	interface FeatureRecord {
2	entityId: string;
3	eventTimeMs: number;
4	version: number;
5	values: Record<string, number>;
6	}
7
8	export function validForLabel(record: FeatureRecord, labelTimeMs: number): boolean {
9	return record.eventTimeMs <= labelTimeMs;
10	}