Design a Data Anomaly Detection Pipeline

Problem statement

Design a pipeline that ingests metrics and log-derived signals from many heterogeneous sources, maintains adaptive baselines per series, scores every observation with streaming and batch detectors, collapses related findings into correlated incident candidates, and routes tunable alerts to dashboards and paging systems while learning from operator feedback. The product promise is simple: catch real problems within seconds to minutes, and do not page a human for noise.

Static thresholds fail this promise. A fixed limit of 100 req/s is wrong at 3 AM, wrong on Black Friday, and wrong five minutes after a deploy. Real traffic has daily and weekly seasonality, trend, holiday effects, deploy step-changes, and missing-data gaps. A credible design therefore separates four planes:

1. Ingestion plane: collectors, scrapers, push protocols, and a durable stream that preserves per-series order and survives detector restarts.
2. Detection plane: stateful streaming detectors for fast anomalies, scheduled batch detectors for seasonal and slow anomalies, and a model registry with versioned configs.
3. Alerting and correlation plane: scoring-to-event conversion, deduplication, temporal and topology-aware grouping, sensitivity policy, routing, escalation, and suppression.
4. Storage and learning plane: TSDB for raw and downsampled data, baseline and model store, anomaly event store, operator feedback store, and a lake for batch retraining.

Why this is a distinctive design problem

The hard part is not naming an algorithm; it is running two million stateful detectors with per-series state, bounded lag, and a false-positive budget, then converting a storm of raw scores into three meaningful incident groups instead of three hundred pages. Detection is a write-heavy streaming problem; correlation is a stateful windowing problem; alerting is a policy and human-capacity problem; baselining is a storage and batch-compute problem. Treating them as one monolith produces either late detection or alert fatigue.

Public evidence versus design assumptions

Public, cited evidence shows the category is real and solved at scale: Twitter open-sourced its AnomalyDetection library implementing Seasonal Hybrid ESD in 2015; LinkedIn open-sourced luminol and later built ThirdEye for anomaly detection and root-cause analysis; Microsoft published its Spectral Residual based time-series anomaly detection service at KDD 2019; Amazon published Robust Random Cut Forest at ICML 2016 and productized it in Kinesis and SageMaker. These are approach citations, not capacity claims for our system.

For capacity planning this answer explicitly assumes: 2,000,000 active metric series, a 10-second default scrape interval, 200,000 points per second average ingest with a 5x provisioned peak, 120-byte average points, 15-day raw retention, and a raw anomaly rate near 0.05% of points before correlation. Unless tied to a citation, every number is a stated design assumption, target, or budget.

The interview posture

Lead with the false-positive budget and the authority chain: scores are cheap, anomalies are claims, alerts are human interruptions. A strong answer quantifies ingest, defines per-series state and recovery, names at least one streaming and one batch algorithm with its failure mode, and shows how one root cause becomes one correlated group.