Design a Data Quality Validation Framework

Problem statement

Design a data quality validation framework that checks incoming or batch data for validation errors and anomalies, then either lets it flow, flags it, quarantines it, or blocks downstream consumption when failure thresholds exceed configured limits. The framework must support rule-based checks (schema conformance, completeness, uniqueness, range, referential integrity, freshness, volume) and ML-based anomaly detection (distribution drift, outlier bursts, seasonal volume deviation), run them in real time on streaming events and batch over lakehouse tables, and integrate with lineage and governance so that a consumer can see exactly which upstream checks a dataset passed.

This is not "clean your data." A strong answer treats data quality as a control plane wrapped around the data plane. The data plane—Kafka topics, Iceberg tables, warehouse schemas—keeps moving bytes. The control plane intercepts at well-defined gates, evaluates a compiled set of expectations, produces a signed verdict, and enforces policy. If the control plane is slow, wrong, or unavailable, bad data silently poisons every downstream model, dashboard, and revenue report.

Why the problem is distinctive

A web backend can return an error and retry. A data pipeline cannot "retry" a week of silently corrupted training data that already shipped a model. The damage is latent and compounding : bad values flow into feature stores, aggregates, and ML models before anyone notices. Therefore the design separates detection from decision from action . Detection answers "is this record/table anomalous?" Decision answers "given severity, ownership, blast radius, and SLA, do we pass, warn, quarantine, or block?" Action answers "who is paged, what is quarantined, what downstream lineage is frozen?"

The attached brief requires rule-based or ML-based checks, real-time or batch scanning for missing fields and outliers, automatic alerts or quarantining of bad data, and lineage/governance integration, while scaling to large volumes quickly, staying resilient when data fails checks, auditing every rejection, and supporting self-healing. The field shapes, multilingual code contract, section rhythm, quiz placement, multi-cloud diagrams, and WinSystemDesign Friend rules follow the orchestrator, master prompt, course profile, and the uploaded gold standard.

Public operating baseline versus design assumptions

Public evidence establishes that this category is operationally real and hard. Uber has described Databook as its metadata and data-quality platform covering millions of datasets and its uDon system for automated dataset anomaly detection that files tickets when time-series metrics deviate. LinkedIn open-sourced DataHub and has spoken about data quality assertions and its Thirdwave alerting platform; DataHub now models quality results as first-class metadata. Netflix runs one of the largest Apache Iceberg deployments and has published on monitoring table health at petabyte scale. Amazon open-sourced deequ , a Spark-based unit-testing library for data that computes completeness, uniqueness, and approximate-quantile analyzers. Monte Carlo popularized the "six pillars" of data observability—freshness, volume, schema, lineage, distribution, and quality. These are cited company signals, not requirements for our fictional system.

For capacity planning, this answer explicitly assumes a mature analytics organization with 5,000 registered datasets, 50,000 active rules, 10 billion records validated per day in batch, 2 million events per second in streaming validation at peak, and a five-times event peak . Unless a number is tied to a citation, it is a stated design assumption, target, budget, or illustrative threshold—not a claim about any company's private architecture.

The five architectural planes

1. Ingestion plane: hooks that capture data at gates—Kafka consumers, CDC taps, Airflow/Dagster sensors, Iceberg commit listeners, schema-registry events.
2. Rule plane: a versioned rule registry, a DSL compiler that turns expectations into vectorized predicates and aggregate queries, and a policy engine that groups rules into suites.
3. Execution plane: streaming validators (Flink) and batch validators (Spark/Iceberg) that evaluate compiled rules with columnar efficiency.
4. Decision plane: scoring, thresholds, circuit breakers, quarantine routing, and the pass/warn/quarantine/block verdict.
5. Observability & governance plane: dashboards, alerts, lineage annotation, SLA tracking, audit logs, and ownership.

A strong interview answer keeps these planes separate. It allows the execution plane to degrade without losing audit evidence, and it lets the rule plane evolve checks without silently changing enforcement semantics for a validated dataset.