Design Message Retention

Section 1: Understanding

Message retention is primarily a deterministic policy system with storage lifecycle enforcement.

Design it as control plane plus data plane: policy authoring and precedence on one side,

expiration, tombstoning, archive export, and purge orchestration on the other.

Core invariants

Legal hold always overrides TTL.
Lifecycle actions are idempotent and replay-safe.
Every retention action emits immutable audit evidence.
Search index visibility must track message lifecycle exactly.
Policy changes are versioned and replayable.

Practical implementation

Store immutable message payload plus mutable lifecycle metadata: retention class, expireAt, holdState, and purgeState. A policy compiler turns human-readable rules into executable matchers with version hash.

javaOne Dark Pro

1 public record RetentionDecision(String policyId, long expireAtEpochMs, boolean legalHold) {}
2 public class RetentionEngine {
3   public RetentionDecision decide(MessageEnvelope m, PolicySnapshot policy) {
4     if (m.legalHold()) return new RetentionDecision(policy.id(), Long.MAX_VALUE, true);
5     long ttlMs = policy.ttlMsFor(m.channelType(), m.classification());
6     return new RetentionDecision(policy.id(), m.createdAtEpochMs() + ttlMs, false);
7   }
8 }

pythonOne Dark Pro

1 from dataclasses import dataclass
2  
3 @dataclass
4 class RetentionDecision:
5     policy_id: str
6     expire_at_ms: int
7     legal_hold: bool
8  
9 def decide_retention(message: dict, policy: dict) -> RetentionDecision:
10     if message.get('legal_hold', False):
11         return RetentionDecision(policy['id'], 2**63 - 1, True)
12     ttl_ms = policy['ttl_ms_by_class'][message['classification']]
13     return RetentionDecision(policy['id'], message['created_at_ms'] + ttl_ms, False)

typescriptOne Dark Pro

1 interface RetentionDecision { policyId: string; expireAtMs: number; legalHold: boolean }
2 export function decideRetention(msg: any, policy: any): RetentionDecision {
3   if (msg.legalHold) return { policyId: policy.id, expireAtMs: Number.MAX_SAFE_INTEGER, legalHold: true };
4   const ttlMs = policy.ttlMsByClass[msg.classification] as number;
5   return { policyId: policy.id, expireAtMs: msg.createdAtMs + ttlMs, legalHold: false };
6 }

Interview framing

State trade-offs clearly: hard delete is simpler conceptually but operationally risky at scale, while mark-and-sweep is safer for retries, canary rollout, and compliance auditing.

Why interviewers care

Message Retention interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

The failure that defines the design

The outage to narrate is deleting data that was under legal hold. A retention policy says delete-after-ninety-days, a bulk sweep runs, and it purges messages that an active litigation hold required you to preserve — spoliation, the cardinal compliance failure, with legal and regulatory consequences far worse than keeping data too long. The fix is the spine of the design: a retention policy engine that resolves each message's effective rule from the applicable policies, with one inviolable precedence — a legal hold always wins and suppresses deletion, no matter what any delete-after rule says — and a sweep that checks the hold before purging anything. State legal-hold-suppresses-deletion-and-is-checked-first up front, because retention is a governance problem where the catastrophic error is not over-retaining but destroying something you were ordered to keep.

Key Highlights

•Deterministic policy semantics
•Idempotent purge and archive flow
•Interview-ready reliability and compliance trade-offs

Section 1: Understanding for Message Retention