Design a Centralized Metadata & Schema Registry

Medium45 min
1 / 30
understanding•10 min read

Problem Statement: A Control Plane for Data Contracts

Frames the registry as the authoritative source of truth for schema definitions, versions, compatibility, and ownership in a large data organization.

Problem statement

Design a centralized metadata and schema registry that stores schema definitions for hundreds to thousands of data streams and tables, tracks every version change, validates compatibility before a new version is accepted, and lets producers and consumers coordinate updates without breaking each other. The product must support CRUD for schemas with immutable versioning, machine-enforced compatibility rules such as "never remove a required field", searchable dataset metadata, and ownership or role-based tagging so that every schema has an accountable team.

In a mature big-data organization, schemas are contracts. A producer team changes a Kafka topic or a warehouse table; dozens of downstream consumers—dashboards, ML features, ETL jobs, reverse-ETL syncs—silently depend on field names, types, and nullability. Without a registry, evolution happens by accident: a field is renamed, a consumer deserializer throws at 3 a.m., and the on-call engineer discovers the contract by reading failing stack traces. The registry turns that chaos into a governed workflow: propose a schema, validate compatibility against history, register an immutable version, and let every serializer resolve a stable identifier.

Why the problem is distinctive

A schema registry is read-dominated, write-governed, and correctness-critical on a tiny dataset. The stored payload is small—tens of thousands of schema texts of a few kilobytes—yet the registry sits on the hot path of every producer serialize and consumer deserialize, and on the critical path of every CI pipeline that ships a data contract. The design therefore separates three planes: a governance write path that is strongly consistent, audited, and leader-serialized; a read path that is cacheable, immutable-by-id, and survivable during registry outage; and a discovery plane—search, lineage, ownership—that can be eventually consistent without endangering serialization.

Public systems prove the category. Confluent Schema Registry documents storing schemas in an internal compacted Kafka topic, serving REST lookups, and enforcing BACKWARD, FORWARD, and FULL compatibility modes with transitive variants. LinkedIn open-sourced DataHub, a metadata platform built on metadata change events over Kafka with Elasticsearch-backed search. Lyft open-sourced Amundsen with a graph-backed catalog and ETL-driven ingestion. Netflix open-sourced Metacat as a federated metadata layer. These are cited architectures, not requirements for our fictional system; every uncited number below is an explicit design assumption.

The three architectural planes

  1. Governance plane: subject and version aggregates, compatibility policy, ownership, approval workflow, audit log.
  2. Serving plane: schema-by-id resolution, latest-version reads, client caches, replication, outage fallback.
  3. Discovery plane: search index, lineage graph, documentation, data-classification tags, governance-tool integration.

A strong interview answer keeps these planes separate, assigns each a different consistency model, and never lets discovery lag corrupt the serialization path.

Key Highlights

  • •The registry is read-dominated and write-governed: hot immutable reads, slow audited writes.
  • •Schema content addressed by id is immutable and cacheable forever; latest-version pointers are the only mutable hot state.
  • •Confluent stores schemas in a compacted internal Kafka topic and enforces named compatibility modes.
  • •Three planes—governance, serving, discovery—each get their own consistency model and failure behavior.
  • •A registry outage must not stop consumers: client-side caches and wire-format schema ids are the safety net.
Lead With Read/Write Asymmetry
State in the first two minutes that the registry is a tiny, correctness-critical dataset with a massive read fan-out. That single framing justifies immutable ids, aggressive caching, and a serialized write path before any box is drawn.
Do Not Make the Registry a Hard Dependency
A design where every deserialize blocks on a registry RPC fails the first network partition. Consumers must cache schema-by-id content and embed the id in the wire format so reads survive registry outage.

Section Rescue Kit

Buzzwords to use:

Schema RegistryCompatibility Mode

Safe statements:

  • "Let me separate the governance write path from the hot read path before choosing storage."
  • "The registry's dataset is small; the engineering problem is correctness under massive read fan-out."
Design a Centralized Metadata & Schema Registry - System Design | WinJob | WinJob