Design Rate Limiter

Problem Statement & Context

A distributed rate limiter decides, in under a millisecond, whether a request proceeds or gets a 429 Too Many Requests. It is the valve Cloudflare, Stripe, and AWS API Gateway put in front of everything: it protects origin capacity from spikes, protects margins from runaway usage, and keeps a multi-tenant API fair so one noisy customer cannot starve the rest. The framing that earns early credibility is the control-plane / data-plane split — policy (who gets how many requests) is authored and distributed relatively slowly, while the counter mutation on each request must stay sub-millisecond at peak.

The problem is deceptively deep because it is a distributed counting problem under latency pressure. A single-node counter is trivial; the hard version keeps counts accurate enough across hundreds of edge nodes and Redis shards without paying a cross-region round-trip per request. That tension — accuracy versus latency — is the spine of the whole design, and almost every later decision (which algorithm, how to shard, where to enforce) is a different answer to it.

Concrete scope

Three workloads anchor the discussion, and each stresses a different axis: token decisions made at the CDN edge before any origin load (latency), multi-tenant SaaS tiers with different burst ceilings (fairness and policy distribution), and Stripe-style payment APIs where a 429 must be idempotency-safe and never double-charge (correctness). Naming these up front signals you understand a rate limiter is not one thing.

Numbers to commit to

State targets early so the design has something to be measured against: a 12M-checks/sec global peak, p99 < 3 ms for a same-region decision, ~50M active keys resident, a < 0.01% false-allow rate, and config propagation in < 5 s. These are the SLOs every subsequent section defends.

Interview checkpoint

Open by separating the control plane (policy authoring and distribution) from the data plane (per-request counter mutation), then name the central tension — accuracy vs latency in a distributed count — and the three workloads that stress it. That framing beats jumping straight to "use a token bucket."