Design a Data Retention & Archival Workflow

Medium45 min
1 / 30
understanding•10 min read

Problem Statement: Data Lifecycle Is a Distributed System, Not a Cron Job

Frames retention and archival as a policy-driven distributed platform with four planes: policy, execution, verification, and governance.

Problem statement

Design a data retention and archival workflow that automatically moves older data from hot, fast, expensive storage to progressively cheaper and slower tiers after a defined age, enforces retention policies, supports rehydration when archived data is needed again, and proves compliance to auditors. The system must detect data age and usage patterns, move or replicate data to cheaper tiers or offline backups, restore or rehydrate on request, and do all of this under explicit, rule-based policy rather than ad-hoc operator decisions.

At small scale this is a cron script that runs aws s3 cp and deletes old files. At platform scale it becomes one of the most dangerous subsystems in the company: it is the only system with standing permission to delete production data at petabyte scale. A bug, a stale policy, or a clock skew can erase data that is under legal hold, required for regulatory retention, or still actively queried. Therefore the design must separate the decision to act from the act itself, and insert verification and reversible windows between them.

Why the problem is distinctive

Most data infrastructure answers questions of speed: how fast can we ingest, query, or serve. Retention answers a different question: how do we safely stop paying for data while remaining able to produce it. Three properties make it distinctive. First, the working set is enormous but the decision rate per object is tiny: an object is evaluated against policy perhaps once per day but read rarely, so metadata management dominates cost, not bandwidth. Second, the blast radius of errors is asymmetric: a missed transition wastes money, but a wrongful deletion is unrecoverable and may be illegal. Third, the system must reconcile three clocks: physical age of the data, business age defined by policy, and legal age defined by holds and regulations, which can override both.

Amazon has publicly described the scale that forces this discipline: S3 stores hundreds of trillions of objects (Amazon disclosed 100 trillion objects in 2021 and more than 350 trillion by 2023), and S3 Lifecycle and Intelligent-Tiering exist precisely because no human can manage placement at that cardinality. Those are public vendor figures used as context; every scale number used for sizing this design is an explicit assumption stated later.

The four architectural planes

  1. Policy plane: versioned retention rules, dataset classification, legal holds, regulatory mappings, and approval workflows.
  2. Execution plane: inventory scanning, transition and deletion jobs, tiered copy engines, and restore orchestration.
  3. Verification plane: checksums, manifests, copy-verify-delete sequencing, reconciliation, and audit evidence.
  4. Governance plane: cost reporting, compliance dashboards, exception workflows, and incident forensics.

A strong answer keeps these planes separate. A failure in the execution plane (a stuck copy job) must never silently change policy semantics, and a policy change must never reach production without a governance gate. This separation is what lets the system delete data at scale while remaining trustworthy.

Key Highlights

  • •Retention is the only subsystem with standing permission to delete production data at petabyte scale; its safety design matters more than its throughput.
  • •Separate the decision to act from the act itself, with verification and reversible windows between them.
  • •Metadata management, not bandwidth, dominates cost when billions of objects are evaluated daily.
  • •Three clocks must reconcile: physical data age, business policy age, and legal hold age.
  • •Four planes: policy, execution, verification, governance. Failures in one must not corrupt another.
Lead With the Danger Model
State in the first two minutes that this system has standing permission to delete production data, so correctness and reversibility outrank throughput. That framing instantly separates a platform design from a script.
Do Not Design a Cron Job
A design where a single scheduler both decides and deletes, with no inventory, no verification, and no hold check, fails under any real compliance or scale probe.

Section Rescue Kit

Buzzwords to use:

Data Lifecycle ManagementTiered Storage

Safe statements:

  • "I will separate the decision to act from the act itself, because deletion at scale must be verifiable and reversible."
  • "Before choosing storage classes, let me define which plane owns policy, execution, verification, and audit."
Design a Data Retention & Archival Workflow - System Design | WinJob | WinJob