Design Key Rotation System - System Design | WinJob

Problem Statement & Key Rotation Mandate

Key points

Keys under management: 50M+
Scheduled rotations/day: 120k
Re-wrap throughput target: 2M blobs/hr

Deep dive

Platform teams at AWS KMS, HashiCorp Vault, and Google Cloud must rotate master keys, data encryption keys (DEKs), JWT signing keys, TLS certificates, and API secrets on policy—often daily for high-risk material—without invalidating in-flight sessions or forcing fleet-wide restarts. Interviewers probe whether you treat rotation as a distributed workflow with versioned ciphertext, not a cron job that swaps one env var. Add operational detail: coordinators remain stateless; durable state lives in regional SQL with optimistic concurrency on rotation_epoch. Re-wrap consumers read keyed partitions so a single enterprise tenant cannot starve shared pools. Publication path invalidates edge caches using ETag bumps tied to version.active events—not wall-clock cron.

javaOne Dark Pro

1 public final class KeyVersionHeader {
2   private final String keyId;
3   private final int version;
4   public byte[] encode() {
5     return (keyId + "|" + version).getBytes(StandardCharsets.UTF_8);
6   }
7 }

pythonOne Dark Pro

1 @dataclass(frozen=True)
2 class RotationPolicy:
3     key_id: str
4     overlap_hours: int
5     rewrap_batch_size: int = 1000

typescriptOne Dark Pro

1 interface VerifySet {
2   activeEncryptKid: string;
3   verifyKids: string[];
4 }
5 export function pickVerifier(kid: string, set: VerifySet): boolean {
6   return set.verifyKids.includes(kid);
7 }

Operational notes

Section 1 note 1: Problem Statement & Key Rotation Mandate — overlap windows for understanding phase.
Section 1 note 2: Problem Statement & Key Rotation Mandate — JWKS publish order for understanding phase.
Section 1 note 3: Problem Statement & Key Rotation Mandate — idempotent re-wrap for understanding phase.
Section 1 note 4: Problem Statement & Key Rotation Mandate — tenant fair queuing for understanding phase.
Section 1 note 5: Problem Statement & Key Rotation Mandate — HSM token buckets for understanding phase.
Section 1 note 6: Problem Statement & Key Rotation Mandate — verify mismatch metrics for understanding phase.
Section 1 note 7: Problem Statement & Key Rotation Mandate — freeze API during incidents for understanding phase.
Section 1 note 8: Problem Statement & Key Rotation Mandate — dual-control emergency rotate for understanding phase.
Section 1 note 9: Problem Statement & Key Rotation Mandate — encryption context on blobs for understanding phase.
Section 1 note 10: Problem Statement & Key Rotation Mandate — workflow cursors in S3 for understanding phase.
Section 1 note 11: Problem Statement & Key Rotation Mandate — RETIRED only after zero decrypts for understanding phase.
Section 1 note 12: Problem Statement & Key Rotation Mandate — correlate traces with rotation_id for understanding phase.

If challenged on blast radius, answer with overlap window + key version in ciphertext header—never “restart all pods at 3am.”

Why interviewers care

Key Rotation System interviews reward crisp scope, explicit trade-offs, and failure stories—not generic microservice diagrams.

Interview checkpoint

Name one failure story for Problem Statement & Key Rotation Mandate that proves you understand real outages, not happy-path diagrams.

Key Highlights

•Keys under management: 50M+
•Scheduled rotations/day: 120k
•Re-wrap throughput target: 2M blobs/hr

1	public final class KeyVersionHeader {
2	private final String keyId;
3	private final int version;
4	public byte[] encode() {
5	return (keyId + "\|" + version).getBytes(StandardCharsets.UTF_8);
6	}
7	}

1	@dataclass(frozen=True)
2	class RotationPolicy:
3	key_id: str
4	overlap_hours: int
5	rewrap_batch_size: int = 1000

1	interface VerifySet {
2	activeEncryptKid: string;
3	verifyKids: string[];
4	}
5	export function pickVerifier(kid: string, set: VerifySet): boolean {
6	return set.verifyKids.includes(kid);
7	}