Design a Stream-Processing ML Feature Extraction

Problem statement

Design a stream-processing ML feature extraction platform that consumes raw event streams—clickstream, transactions, sensor readings, fraud signals—and computes features in real time so an ML model can score requests with up-to-date context. Features include rolling counts (transactions in the last 5 minutes), rolling sums and averages (spend in the last 24 hours), session statistics (items in current session), and recency features (time since last purchase). The platform must update these features in an online store readable at sub-10 ms latency, and also materialize them to an offline store so training data can be reconstructed with point-in-time correctness.

This is not a generic stream-processing question. The distinguishing tension is training-serving skew: the exact same feature logic must produce identical values whether computed online over a live stream or offline over historical data for model training. If the batch pipeline computes avg_order_value_7d differently from the streaming pipeline—even by rounding mode—the model trains on one distribution and scores against another, silently degrading accuracy.

A second distinguishing tension is correctness under failure. A feature update is often a read-modify-write cycle: read current counter, add event delta, write new value. If the processor crashes between read and write, or processes an event twice after recovery, the feature is corrupted. At 500K events/sec with 100M entity keys, this is not a theoretical race condition—it is the primary engineering problem.

Why this problem is distinctive

A traditional ETL pipeline can retry a failed batch job overnight. A streaming feature pipeline cannot pause while a fraud model waits for the txn_count_last_5m feature. The design therefore separates feature computation (windowed aggregation over event streams) from feature serving (low-latency key-value reads) from feature governance (registry, versioning, lineage, monitoring). Each plane has different consistency requirements, failure semantics, and scaling characteristics.

The brief requires streaming ingestion with ephemeral state management, rolling metric computation, feature store updates or immediate inference, and time-window logic for recency-based features. Non-functional requirements include low-latency generation, scalability for large event volumes, data correctness avoiding partial updates, and exactly-once or idempotent handling.

The four architectural planes

1. Ingestion plane: durable event backbone, schema enforcement, partitioning by entity key.
2. Computation plane: stateful stream processor, windowed aggregations, watermarks, checkpoints.
3. Serving plane: online store for sub-ms reads, offline store for training, point-in-time join engine.
4. Governance plane: feature registry, versioning, lineage, drift monitoring, access control.

A strong answer keeps these planes separate. It allows the serving plane to degrade (serve stale features) without corrupting the computation plane, and it permits backfilling new features without disrupting live inference.