Design a Rolling Weighted DNS Update

Medium45 min
1 / 30
System design answer

Design a Rolling Weighted DNS Update

Medium45 min30 sections
understanding•10 min read

Problem Statement: Progressive Delivery at the Name-Resolution Layer

Frames weighted DNS migration as a cache-dominated, eventually consistent control problem rather than a simple record edit.

Problem statement

Design a platform that performs a rolling weighted DNS update for a production domain: traffic is gradually shifted from an old endpoint (origin IP, load balancer, or region) to a new endpoint by adjusting weighted DNS records, while error rates, latency, and health signals are watched continuously, and the change is reverted automatically or manually if the new endpoint degrades. The classic shape is: start at 100/0 old/new, step through 99/1, 95/5, 90/10, 75/25, 50/50, 25/75, 10/90, 0/100, dwell at each step until evidence says the new endpoint is safe, and complete or roll back.

The problem requires weighted DNS records or route policies, gradual increments toward the new IPs, TTL configuration for fast propagation with controlled caching, and automated rollback when metrics degrade. Those four requirements look small. The difficulty is that DNS is the most widely deployed, most aggressively cached, least controllable load balancer on the internet, and a rolling weighted update is a distributed-systems problem disguised as a config change.

Why DNS is a different beast than a load balancer canary

With an L7 load balancer or service mesh, the shifting component sits in the request path: it decides per request, observes every response, and can reverse a decision in milliseconds. Weighted DNS operates before any request exists. A resolver asks the authoritative servers, receives one or more records with weights, picks among them probabilistically, and then caches the answer for the record TTL. From that moment, the authoritative servers have no visibility into how many clients use the cached answer, no ability to revoke it, and no way to accelerate its expiry. Three consequences follow:

  1. Shift speed is bounded by TTL and cache behavior, not by API speed. Publishing a new weight takes seconds; making the internet converge on it takes minutes to hours depending on TTL, resolver compliance, and client connection pinning.
  2. Rollback is equally slow. If the new endpoint starts failing at 50% weight, setting the weight back to zero stops new resolutions immediately, but every resolver still holding the cached answer keeps sending traffic until that answer expires. The blast radius of a bad step is bounded below by roughly one TTL of continued bad traffic, and in practice more because some resolvers ignore or floor TTLs.
  3. Observation is indirect. The DNS layer cannot tell you the realized traffic split. You reconstruct it from server-side signals: which endpoint served each request, tagged via server identity, edge logs, or connection metadata.

The four architectural planes

A strong answer separates the system into four planes with different consistency and latency characteristics:

  • Control plane: shift plans, desired weights, step schedules, approvals, guardrails. Strongly consistent, single writer per domain, low QPS, high stakes.
  • Publishing plane: provider adapters that translate desired state into Route 53 weighted records, Cloud DNS routing policies, or Traffic Manager profiles, and reconcile published state back. Eventual consistency bounded by provider propagation.
  • Observation plane: health probes, synthetic checks, endpoint metrics, realized-split estimation. High volume, eventually consistent, lossy by design.
  • Decision plane: the automation that compares observations against guardrails and decides step, dwell, pause, or rollback. This plane must remain correct when the control plane is down, which means it needs independent credentials and its own evidence.

Real operating context

This is not a hypothetical mechanism. AWS Route 53 has offered weighted routing since the early 2010s with weights from 0 to 255 per record set and optional health-check association; Azure Traffic Manager has weighted traffic routing with weights from 0 to 1000; Google Cloud DNS added weighted routing policies in 2023; Cloudflare Load Balancing steers across weighted pools with health monitors; and NS1, now part of IBM, built an entire traffic-steering product on weighted pools and filter chains. Verisign's Domain Name Industry Brief reports well over 360 million registered domain names, all ultimately served through the 13 logical root server names and their roughly 1,800 anycast instances. The mechanism is mature; the engineering discipline around it is what interviews probe.

Every uncited number in this answer is an explicit design assumption, stated where it is used. Public company figures are labeled as such.

Key Highlights

  • •Weighted DNS shifts traffic at the resolution layer, before any request exists, so the system cannot observe or revoke cached answers.
  • •Shift and rollback speed are bounded by TTL, resolver compliance, and client connection pinning, not by API latency.
  • •The architecture splits into control, publishing, observation, and decision planes with different consistency models.
  • •Route 53 (weights 0-255), Traffic Manager (0-1000), Cloud DNS routing policies, and Cloudflare pools all implement the same primitive differently.
  • •The realized traffic split must be reconstructed from server-side signals because authoritative DNS never sees cached reuse.
Lead With the Cache Constraint
State in the first two minutes that propagation is bounded by TTL and resolver behavior, not by how fast you can call the DNS API. That single sentence reframes the whole design and separates a DNS migration answer from a load balancer answer.
Do Not Treat It as a Config Edit
A design that draws one engineer editing records and one dashboard misses the core problem: an eventually consistent global cache sits between you and every user, and your rollback has to travel through it too.

Section Rescue Kit

Buzzwords to use:

Cache ResidencyRealized Split

Safe statements:

  • "I will separate how fast we can publish weights from how fast the internet converges on them."
  • "Before choosing a provider, let me define which decisions live in the control plane and which must survive its failure."
Design a Rolling Weighted DNS Update - System Design | WinJob | WinJob