Problem Statement: Privacy Engineering as a Distributed System
Frames the masking service as a policy-enforcement platform spanning discovery, transformation, governance, and audit — not a regex filter.
Problem statement
Design a data privacy and compliance masking service for a large enterprise data platform. The service must find sensitive data wherever it lands — data lake, warehouse, streaming topics, operational databases, BI extracts — classify it against a regulated taxonomy, apply the correct protective transformation (masking, tokenization, pseudonymization, encryption, generalization), allow authorized unmasking only in privileged contexts, and produce regulator-grade evidence for every decision. The brief requires data scanning for sensitive fields, policies for masking/tokenizing/encrypting, role- or context-based partial unmasking, auditing of every unmasking request, scalability for large ingestion pipelines, controlled performance overhead, compliance logs, and integration with data governance.
This is not a string-replacement problem. A masking rule that edits one table is trivial; a masking system that stays correct across 40,000 datasets, 1.2 billion columns, nightly pipeline fan-out, derived tables, ML feature stores, and ad-hoc analyst copies is a distributed-systems problem with a legal contract attached. The hard part is not transforming a value once; it is guaranteeing that every copy, derivative, and downstream join of that value inherits the correct protection, and that no path exists from an unauthorized principal to plaintext.
Why the problem is distinctive
A generic access-control system says no. A privacy platform must say no while still letting analytics happen. Masked data must remain joinable, groupable, schema-compatible, and statistically plausible, or the organization simply routes around the platform. That creates three irreducible tensions: utility versus protection, reversibility versus breach blast-radius, and throughput versus per-cell policy evaluation. Every architecture decision in this answer is a position on one of those tensions.
The four architectural planes
- Discovery plane: connectors, sampling scanners, detectors, ML classifiers, confidence scoring, review queues, and the classification label store.
- Transformation plane: policy engine, transformation library (mask, token, hash, shift, generalize, null), batch masking jobs, materialized masked views, and the inline masking gateway.
- Governance plane: policy authoring and versioning, policy bindings, role and purpose evaluation, unmask grants, erasure orchestration, and lineage-driven propagation.
- Evidence plane: append-only audit trail, decision logs, release attestations, regulator export, and metrics that prove the control operated.
A strong answer keeps these planes separate. Discovery can be eventually consistent and probabilistic; token vault and grant decisions must be strongly consistent and deterministic; audit must be immutable. Collapsing them into one database with one consistency level is the most common design failure.
Public operating baseline versus design assumptions
The category is operationally real. Google offers Sensitive Data Protection (formerly Cloud DLP) with built-in infoType detectors and de-identification transforms including format-preserving encryption; AWS offers Macie for S3 discovery plus Lake Formation cell-based security; Microsoft offers Purview sensitivity labels plus Azure SQL dynamic data masking. These are cited public product capabilities used as context. For capacity planning this answer explicitly assumes a mature enterprise with a 250 PB governed lake, 60 TB of new or changed data per day, 15,000 masking-gateway queries per second average, and 2 billion active tokens. Unless a number is tied to a citation, it is a stated design assumption, target, or budget — not a claim about any company's private deployment.
Key Highlights
- •The service spans four planes: discovery, transformation, governance, and evidence — each with different consistency and latency contracts.
- •Correctness means every copy and derivative of a sensitive value inherits protection, not just the source table.
- •Three tensions drive every decision: utility vs protection, reversibility vs blast radius, throughput vs per-cell policy.
- •Discovery may be probabilistic; vault and grants must be deterministic; audit must be immutable.
- •All uncited scale numbers are explicit design assumptions, labeled as such.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate discovery, enforcement, governance, and evidence, because each has a different consistency and failure contract."
- "Before picking transformations, let me define what no principal may ever be able to reach: plaintext without a grant."