Design a Data Lake (Hadoop/S3)

Problem statement

Design a data lake that stores raw structured, semi-structured, and unstructured data—application logs, clickstreams, CSV extracts, JSON events, images, and database change captures—at petabyte scale, so that analysts, data scientists, and ML pipelines can query it later with schema-on-read engines such as Hive, Presto/Trino, and Spark. The lake must accept batch and streaming ingestion, expose a catalog for discovery, enforce retention and lifecycle policies, and remain durable and cost-efficient while integrating with every major compute engine.

A data lake is not a big folder. It is a contract between four planes. The ingestion plane lands data immutably and records exactly what arrived. The storage and metadata plane keeps bytes cheap and durable while keeping files discoverable through partitions, manifests, statistics, and a catalog. The governance plane decides who may read or write which dataset, for how long, and with which encryption and audit rules. The consumption plane lets engines prune petabytes down to gigabytes before reading a single byte. A strong interview answer designs all four planes and states the consistency model of each.

Why the problem is distinctive

A transactional database optimizes one row at a time; a data lake optimizes one scan at a time. The unit of design is therefore the file, the partition, and the metadata commit—not the row. The three classic failure modes are the small-files problem (millions of 1 MB objects destroy listing and task scheduling), the metadata bottleneck (a single NameNode or metastore becomes the RAM and lock ceiling), and the silent swamp (data no one can find, trust, or delete). Every design decision below attacks one of these three.

HDFS versus object storage framing

Hadoop HDFS gives atomic rename, append semantics, and a POSIX-like namespace, but its NameNode holds every file and block in heap—roughly 150-200 bytes per object—so one billion files demands hundreds of gigabytes of NameNode RAM and federation. Amazon S3 removes the metadata ceiling entirely: AWS publicly states S3 stores trillions of objects with 99.999999999% durability and has provided strong read-after-write and list consistency since December 2020. The trade moves from capacity planning a metadata master to managing request-rate limits per prefix (AWS documents 3,500 PUT/COPY/POST and 5,500 GET/HEAD per second per prefix) and accepting that object stores historically lacked rename, which table formats now solve with metadata-level commits.

Public operating baseline versus design assumptions

Public evidence makes the category concrete: Yahoo ran tens of thousands of Hadoop nodes and hundreds of petabytes of HDFS in the early 2010s; Meta published f4 warm BLOB storage and HDFS erasure coding at exabyte scale; Airbnb publicly described hundreds of petabytes in S3 behind a Hive metastore; Netflix created and open-sourced Apache Iceberg on S3; Uber created Apache Hudi for incremental upserts. These are cited company figures, not requirements for our design. For capacity planning this answer explicitly assumes 600 TB/day raw ingestion (500 TB batch, 100 TB streaming), 90-day raw retention, 5-year curated retention, 20,000 queries per day at a peak of 5 per second, and 50,000 cataloged tables. Unless a number is tied to a citation, it is a stated assumption.

The four architectural planes

1. Ingestion plane: batch loaders, streaming committers, schema validation, quarantine, idempotent commits.
2. Storage and metadata plane: object store or HDFS, file formats, partitioning, compaction, catalog, table-format snapshots.
3. Governance plane: identity, fine-grained authorization, encryption, retention, Object Lock, audit, lineage.
4. Consumption plane: SQL engines, Spark jobs, ML readers, caching, result governance.

Keeping these planes separate lets the lake degrade gracefully: a catalog outage may slow discovery without blocking raw landing, and a query-engine outage must never threaten durability.