Design Backup and Recovery - System Design | WinJob

Problem Statement: Petabyte Backup and Recovery Mandate

Context for section 1 (understanding)

Enterprise backup platforms at Veeam, Rubrik, and AWS Backup scale must treat incremental-forever and ransomware canary files as first-class contracts—not optional UI toggles. This section anchors how architects explain problem statement: petabyte backup and recovery mandate when petabyte fleets (2 PB logical) face ransomware and regional loss simultaneously.

Mechanisms you should articulate

Policy engine binds RPO 15m / RTO 30m to asset classes; violations surface in compliance dashboards before audits fail.
Data path VM → Agent → Gateway → Dedup → Vault enforces checksum-sealed chunks; dedup ratio targets 4.2:1 with block-level change tracking to shrink incremental forever cost.
Control plane catalogs restore points with immutable metadata; restores reference backup window throttling without mutating sealed vault objects.

Operational invariants

Immutability wins over convenience — deletes are legal-hold or lifecycle only; ransomware cannot shorten retention via API.
Restore drills are production events — quarterly vault tamper alerts exercises measure real RTO, not spreadsheet estimates.
Catalog truth is independent of payload — losing index nodes must not destroy vault bits; rebuild index from sealed manifests.

Deep dive (understanding focus)

Interviewers probe how Problem Statement reduces blast radius: isolate tenants, cap ingest bandwidth, and prove incremental-forever with measurable SLIs (job success %, restore verification pass rate, immutable copy age).

Quantify agents per vCPU, WAN Mbps per site, and catalog QPS for search-before-restore. Mention SOC2 evidence: who triggered restore, which snapshot, which approver.

Architecture signals

Draw VM → Agent → Gateway → Dedup → Vault with explicit failure arrows: agent crash → job retry; gateway overload → throttle; vault lock → read-only restore from replica.

Compare hot cache for instant recovery vs cold tier for compliance archive; never conflate them in the same SLA sentence.

Edge cases

Split-brain catalog shard: freeze restores, continue sealed writes with quorum.
Application-consistent hook timeout: fail job loudly; do not mark success with crash-consistent only unless policy allows.
Cross-region lag > RPO: escalate policy breach and block marketing "protected" badges.

Metrics snapshot

Metric	Target	Notes
Backup success	99.95%	Excludes user maintenance windows
Restore verify	100% sampled	Hash compare manifest vs restored VM
Immutable age	≥ policy	Object lock / tape WORM
Drill RTO	≤ 30m	Measured end-to-end

javaOne Dark Pro

1 // Policy evaluation gate (conceptual)
2 public final class BackupPolicyGate {
3   private final Duration rpo;
4   public boolean isBreach(Instant lastSuccess, Instant now) {
5     return Duration.between(lastSuccess, now).compareTo(rpo) > 0;
6   }
7 }

pythonOne Dark Pro

1 def chunk_fingerprint(data: bytes) -> str:
2     import hashlib
3     return hashlib.sha256(data).hexdigest()

How to open this one

The framing that signals depth on petabyte backup is that backups are worthless until a restore is proven: you design around RPO (how much data you can lose) and RTO (how fast you recover), and you test restores continuously because an untested backup is a hope, not a guarantee. Lead with incremental-forever plus periodic synthetic fulls to bound storage, and the failure story that matters: a restore is attempted during a real outage and the backup turns out corrupt or incomplete. That shows you understand recovery, not backup, is the actual deliverable.

Key Highlights

•Policy-bound RPO/RTO per asset class
•Immutable vault independent of catalog
•Dedup chunk store with verification
•Restore drills measure real RTO

1	// Policy evaluation gate (conceptual)
2	public final class BackupPolicyGate {
3	private final Duration rpo;
4	public boolean isBreach(Instant lastSuccess, Instant now) {
5	return Duration.between(lastSuccess, now).compareTo(rpo) > 0;
6	}
7	}

1	def chunk_fingerprint(data: bytes) -> str:
2	import hashlib
3	return hashlib.sha256(data).hexdigest()