Problem Statement: A Petabyte-Scale Genomics Analysis Platform
Frames the system as a durable, consent-aware scientific data platform rather than a generic file-processing queue.
Problem statement
Design a data pipeline platform that ingests raw DNA sequencing data (FASTQ files, gigabytes per sample), stores it durably, runs alignment and variant calling in HPC or distributed compute environments, tracks sample metadata (patient IDs, study, consent, tissue, library preparation), and serves results through specialized genomic indexes so researchers and clinicians can query variants quickly. The platform must support whole-genome sequencing (WGS), whole-exome sequencing (WES), and targeted panels, and it must survive the realities of scientific computing: multi-hour jobs, spot preemption, petabyte archives, reference-genome versioning, and strict medical privacy rules.
This is not a log-processing pipeline. A typical 30x whole human genome produces roughly 100 GB of compressed FASTQ, 25-35 GB of aligned CRAM, and 1-3 GB of gVCF before a single base is interpreted. The human reference genome GRCh38 is about 3.1 billion base pairs, and a single nucleotide error in metadata can mean a result attributed to the wrong patient. The pipeline therefore separates five concerns that generic big-data designs blur: file integrity, scientific reproducibility, compute orchestration, metadata correctness, and consent-governed access.
Why the problem is distinctive
A clickstream pipeline can drop or reprocess events with modest consequences. A genomics pipeline cannot silently swap samples, recompute results against the wrong reference build (GRCh37 versus GRCh38), or expose controlled-access data to an unauthorized cohort. The design therefore treats analysis artifacts as immutable, content-addressed objects identified by checksums, and it treats every workflow run as a durable ledger that pins the exact reference genome, tool versions, container images, and interval lists used. If a result cannot be reproduced from its recorded inputs, the platform has failed even when the variant calls are statistically correct.
The brief requires large file ingestion (FR1), HPC or distributed analysis steps such as alignment and variant detection (FR2), sample metadata tracking (FR3), and specialized indexes for fast queries (FR4), with high compute demand (NFR1), scalability to thousands of samples (NFR2), security and privacy for medical data (NFR3), and workflow orchestration with tools like Cromwell or Nextflow (NFR4). The section rhythm, visual vocabulary, recap contract, and quiz placement follow the attached course contract and gold standard.
Public operating baseline versus design assumptions
Public evidence establishes that this category operates at extreme scale. The 1000 Genomes Project published 2,504 genomes; the ICGC/TCGA Pan-Cancer Analysis of Whole Genomes (PCAWG) standardized 2,792 whole genomes across cloud platforms; UK Biobank enrolled roughly 500,000 participants and has released exome data for approximately 470,000 of them with whole-genome sequencing of the full cohort underway; gnomAD v4 reports roughly 730,000 exomes plus 76,000 genomes; and NCBI's Sequence Read Archive holds tens of petabytes. These are cited public figures used as context, not requirements for our fictional system.
For capacity planning, this answer explicitly assumes a mid-scale population program: 500 WGS samples per day steady state, 1,500 per day peak, a 250,000-sample archive over the program lifetime, and 30x coverage as the default assay. Unless a number is tied to a citation, it is a stated design assumption, target, budget, or illustrative threshold.
The four architectural planes
- Ingestion plane: signed multipart upload, per-part checksums, manifest reconciliation, virus and format validation, and staging into the data lake.
- Compute plane: HPC or cloud batch execution of alignment, deduplication, recalibration, variant calling, joint genotyping, and annotation, with scatter-gather parallelism and spot-aware retries.
- Metadata and consent plane: study, donor, sample, run, consent, and workflow ledgers with strong consistency and purpose-bound access.
- Query and learning plane: tabix/CSI indexed VCFs, columnar cohort stores (TileDB-VCF or Hail), htsget streaming, cohort analytics, and QC dashboards.
A strong interview answer keeps these planes separate. Ingestion failures must not corrupt metadata, compute retries must not duplicate artifacts, and analytics must never read a sample whose consent has been withdrawn without going through the consent gate.
Key Highlights
- •A 30x whole genome is roughly 100 GB FASTQ, 30 GB CRAM, and 2 GB gVCF; bytes dominate but correctness dominates risk.
- •Artifacts are immutable and content-addressed; every workflow run pins exact reference, tool, and container versions.
- •Public figures such as PCAWG's 2,792 genomes and UK Biobank's ~500,000 participants provide context; every other scale number here is an explicit assumption.
- •The architecture has four planes: ingestion, compute, metadata/consent, and query/learning.
- •A result that cannot be reproduced from its recorded inputs is a platform failure even if the biology is right.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate file movement from scientific computation from consent governance, because each fails differently."
- "Before selecting storage classes, let me define which artifacts are immutable inputs, which are derived, and which are disposable intermediates."