Design a Rolling Key Rotation in Production

Medium45 min
1 / 29
System design answer

Design a Rolling Key Rotation in Production

Medium45 min29 sections
understanding•10 min read

Problem Statement: Rotating Live Cryptographic Keys Without Breaking Anyone

Frames rolling key rotation as a four-plane coordination problem across control, data-plane crypto, bulk migration, and audit.

Problem statement

Design a rolling key rotation platform that lets a production fleet rotate encryption keys — for data-at-rest envelopes, session tokens, signing keys, and API HMAC secrets — without downtime, without invalidating active sessions, and without ever exposing key material. Rotation must be staged: a new key version is generated, both old and new remain valid inside a dual-key window, traffic shifts gradually, long-lived ciphertext is re-encrypted in the background, and the old key is retired only after a provable grace period.

This is not a cron job that swaps a config file. The hard part is that at any instant during rotation there are three populations of ciphertext in flight: records encrypted under the old key version, records encrypted under the new key version, and tokens issued minutes ago that must validate under whichever key signed them. A naive cutover decrypts the old population as garbage and rejects every live session. A naive dual-read doubles KMS load and leaves a forever-dependency on the old key. The design must make the key version explicit in every ciphertext and token, keep a version-aware decrypt path, and drive the whole transition as a durable, resumable state machine that survives orchestrator restarts and partial failures.

Why the problem is distinctive

A blue-green deployment can roll back by switching traffic. A key rotation cannot simply "switch back" once data has been re-encrypted under the new key and the old key's grace period has expired — cryptographic transitions are only partially reversible. The design therefore separates rotation correctness (eventually all ciphertext under the newest key, old key destroyed per policy) from service availability (every request succeeds throughout). Availability is the invariant that must never be traded: if the migration pipeline stalls, sessions keep validating; if the new key is suspected compromised, containment beats schedule.

Public operating baseline versus design assumptions

Public evidence establishes that managed rotation is an operational norm. AWS KMS supports automatic annual rotation for symmetric KMS keys, and Google Cloud KMS documents configurable automatic rotation with a minimum period of 90 days for software keys. HashiCorp Vault popularized dynamic secrets with built-in lease expiry and revocation, and Azure Key Vault ships rotation policies with expiry and notification triggers. PCI-DSS has required periodic key rotation and documented key ceremonies since its earliest versions, and NIST SP 800-57 formalized the key lifecycle: pre-activation, active, deactivation (processing), compromised, and destroyed. These are cited industry baselines, not requirements for our fictional system.

For capacity planning this answer explicitly assumes a mature platform: 500 services, 2,000 instances, 50 million encrypted records (~50 GB ciphertext), 100,000 token validations per second at peak, 90-day rotation period, 30-day dual-key window, and 7-day rollback window. Unless a number is tied to a provider's published behavior, it is a stated design assumption, target, or budget.

The four architectural planes

  1. Control plane: rotation policy engine, scheduling, the rotation state machine, approvals, and kill switches.
  2. Crypto data plane: version-aware encrypt/decrypt, envelope encryption, token signing and validation, local key caches.
  3. Migration plane: batch re-encryption workers, checkpoints, backfill, and verification.
  4. Audit plane: immutable key lifecycle events, access logs, compliance evidence, and who-triggered-what records.

A strong interview answer keeps these planes separate. The migration plane may stall without affecting the crypto data plane; the control plane may crash and resume from durable state; the audit plane records every transition whether or not anyone is watching.

Key Highlights

  • •Rotation is a durable state machine, not a config swap: three ciphertext populations coexist during every transition.
  • •Cryptographic transitions are only partially reversible — plan rollback windows before generating the new key.
  • •AWS KMS automatic rotation and Google Cloud KMS 90-day minimum rotation period are published industry baselines.
  • •The architecture has four planes: control, crypto data plane, migration, and audit.
  • •Availability is the invariant that is never traded: migration may stall, sessions must keep validating.
Lead With the Three Populations
State in the first two minutes that during rotation three ciphertext populations coexist (old-key, new-key, in-flight tokens). This instantly distinguishes a real rotation design from a config-swap answer.
Do Not Describe a Big-Bang Swap
A design that flips a flag and swaps keys atomically breaks every old ciphertext and every live token signed under the previous key. Interviewers probe exactly this.

Section Rescue Kit

Buzzwords to use:

Dual-Key WindowEnvelope Encryption

Safe statements:

  • "I will separate rotation correctness from service availability: the migration may stall, but decrypt and validate must never fail."
  • "Before choosing services, let me define which decisions happen in the control plane, which in the crypto data plane, and which require human approval."
Design a Rolling Key Rotation in Production - System Design | WinJob | WinJob