Problem Statement: Rotating Live Cryptographic Keys Without Breaking Anyone
Frames rolling key rotation as a four-plane coordination problem across control, data-plane crypto, bulk migration, and audit.
Problem statement
Design a rolling key rotation platform that lets a production fleet rotate encryption keys — for data-at-rest envelopes, session tokens, signing keys, and API HMAC secrets — without downtime, without invalidating active sessions, and without ever exposing key material. Rotation must be staged: a new key version is generated, both old and new remain valid inside a dual-key window, traffic shifts gradually, long-lived ciphertext is re-encrypted in the background, and the old key is retired only after a provable grace period.
This is not a cron job that swaps a config file. The hard part is that at any instant during rotation there are three populations of ciphertext in flight: records encrypted under the old key version, records encrypted under the new key version, and tokens issued minutes ago that must validate under whichever key signed them. A naive cutover decrypts the old population as garbage and rejects every live session. A naive dual-read doubles KMS load and leaves a forever-dependency on the old key. The design must make the key version explicit in every ciphertext and token, keep a version-aware decrypt path, and drive the whole transition as a durable, resumable state machine that survives orchestrator restarts and partial failures.
Why the problem is distinctive
A blue-green deployment can roll back by switching traffic. A key rotation cannot simply "switch back" once data has been re-encrypted under the new key and the old key's grace period has expired — cryptographic transitions are only partially reversible. The design therefore separates rotation correctness (eventually all ciphertext under the newest key, old key destroyed per policy) from service availability (every request succeeds throughout). Availability is the invariant that must never be traded: if the migration pipeline stalls, sessions keep validating; if the new key is suspected compromised, containment beats schedule.
Public operating baseline versus design assumptions
Public evidence establishes that managed rotation is an operational norm. AWS KMS supports automatic annual rotation for symmetric KMS keys, and Google Cloud KMS documents configurable automatic rotation with a minimum period of 90 days for software keys. HashiCorp Vault popularized dynamic secrets with built-in lease expiry and revocation, and Azure Key Vault ships rotation policies with expiry and notification triggers. PCI-DSS has required periodic key rotation and documented key ceremonies since its earliest versions, and NIST SP 800-57 formalized the key lifecycle: pre-activation, active, deactivation (processing), compromised, and destroyed. These are cited industry baselines, not requirements for our fictional system.
For capacity planning this answer explicitly assumes a mature platform: 500 services, 2,000 instances, 50 million encrypted records (~50 GB ciphertext), 100,000 token validations per second at peak, 90-day rotation period, 30-day dual-key window, and 7-day rollback window. Unless a number is tied to a provider's published behavior, it is a stated design assumption, target, or budget.
The four architectural planes
- Control plane: rotation policy engine, scheduling, the rotation state machine, approvals, and kill switches.
- Crypto data plane: version-aware encrypt/decrypt, envelope encryption, token signing and validation, local key caches.
- Migration plane: batch re-encryption workers, checkpoints, backfill, and verification.
- Audit plane: immutable key lifecycle events, access logs, compliance evidence, and who-triggered-what records.
A strong interview answer keeps these planes separate. The migration plane may stall without affecting the crypto data plane; the control plane may crash and resume from durable state; the audit plane records every transition whether or not anyone is watching.
Key Highlights
- •Rotation is a durable state machine, not a config swap: three ciphertext populations coexist during every transition.
- •Cryptographic transitions are only partially reversible — plan rollback windows before generating the new key.
- •AWS KMS automatic rotation and Google Cloud KMS 90-day minimum rotation period are published industry baselines.
- •The architecture has four planes: control, crypto data plane, migration, and audit.
- •Availability is the invariant that is never traded: migration may stall, sessions must keep validating.
Section Rescue Kit
Buzzwords to use:
Safe statements:
- "I will separate rotation correctness from service availability: the migration may stall, but decrypt and validate must never fail."
- "Before choosing services, let me define which decisions happen in the control plane, which in the crypto data plane, and which require human approval."