Design a Data Governance & Lineage System

Problem statement

Design a data governance and lineage platform for a large organization. It must catalog every dataset, table, stream, dashboard, and ML feature with schema and ownership; automatically capture lineage as pipelines run; enforce role-based and attribute-based access policies; satisfy GDPR/HIPAA obligations including deletion workflows and audit records; and expose a graph API and UI to traverse how data flows from source systems to reports.

The problem looks like a catalog app until you count the workload. A mature data organization does not have one writer updating rows; it has thousands of pipelines emitting schema changes, run events, column projections, and classification results continuously. The catalog is a convergent projection of an event stream, and lineage is a graph assembled from partial, overlapping, sometimes contradictory evidence emitted by heterogeneous engines: Airflow DAGs, Spark plans, dbt manifests, BI tools, hand-written SQL, and streaming jobs.

Why this is distinctive

Three properties separate this from a generic CRUD service. First, write volume comes from machines, not humans: every pipeline run emits lineage and schema events, so ingestion is a streaming problem with idempotency requirements. Second, the core query shape is graph traversal (impact analysis, root-cause tracing, deletion blast radius), not point reads. Third, governance decisions must be fail-closed for sensitive data: a policy service outage must deny access to PII, not accidentally permit it.

Public systems prove the category. LinkedIn open-sourced DataHub, describing a stream-oriented metadata architecture built on entities, aspects, relationships, and Kafka-based change events after its earlier batch-era WhereHows system could not keep up. Uber built Databook as a metadata platform for discovery, governance, and lineage across its data ecosystem. Airbnb built Dataportal for discovery and stewardship, and Lyft created Amundsen, which pairs a graph database with a search index for people-and-data discovery.

Scale baseline for this answer

All uncited numbers below are explicit design assumptions for a large enterprise, stated so the capacity math is reproducible: 250,000 registered data assets (150K tables and views, 20K streams, 50K dashboards and reports, 30K ML features); 8,000 pipelines producing about 48,000 runs per day including retries; roughly 3 million metadata events per day (lineage, schema, ownership, classification); a lineage graph of about 350K dataset nodes, 12M dataset-level edges, and 40M column-level edges; 8,000 policy decisions per second average with a 5x peak; and about 12,000 daily active data practitioners using search and lineage UIs.

The four planes

1. Ingestion plane: OpenLineage-compatible gateways, SQL and plan parsers, schema differ, enrichment, dedup.
2. Graph plane: lineage storage, traversal, impact analysis, convergence bookkeeping.
3. Governance plane: policies, ABAC decisions, glossary terms, ownership, certification, stewardship tasks.
4. Consumption plane: search, lineage UI, APIs, notifications, exports, and audit surfaces.

A strong answer keeps these planes separate so a search outage never blocks policy decisions, and a parser regression never corrupts the audit trail.