Design a Multi-Region Data Replication Strategy

Problem statement

Design a system where data written in multiple geographic regions is replicated to other regions so that analytics jobs, regional services, and disaster-recovery procedures can read it with bounded staleness, bounded loss, and a consistency model stated explicitly per dataset. Data originates in three primary regions (us-east, eu-west, ap-southeast), is produced by application writes, change-data-capture from transactional stores, and streaming events, and must be available for cross-region analytics within a lag budget while respecting data-residency rules.

This is not a backup question. A backup answers 'can we restore yesterday?' Multi-region replication answers 'can us-east-2 serve a write-path read 800 milliseconds after us-east-1 committed it, while eu-west is partitioned, and can we prove nothing was lost or silently reordered?' The hard part is that the network between continents is slow, expensive, and fails. The speed of light imposes a floor: one-way fiber latency between US East and EU West is roughly 70 ms, US East to Singapore roughly 120 ms. Any synchronous replication protocol must pay that round trip on the commit path. Any asynchronous protocol must define what happens when the sender dies before the network drains its buffer.

Why this problem is distinctive

Single-region systems can pretend the network is reliable and fast. Multi-region systems cannot. Three properties make the problem hard. First, writes arrive at different regions at different times, so two regions can receive conflicting updates to the same key without either being wrong. Second, links between regions fail asymmetrically: us-east can push to eu-west while eu-west cannot push back, which creates divergence, not just delay. Third, volume is enormous: an analytics pipeline that ingests 2 billion events per day at 1 KB each generates 2 TB/day of new data, and full-mesh replication across three regions doubles the cross-region transfer to roughly 4 TB/day, so topology choice is a cost decision, not just a correctness decision.

The three architectural axes

1. Topology axis: single-writer (one region owns writes, others follow), multi-active (every region writes, conflicts are resolved), or leaderless quorum (writes go to N replicas, R and W overlap). The brief's requirement FR3 — conflict resolution or versioning if multi-master — means we cannot hand-wave this choice.
2. Synchrony axis: synchronous replication (commit blocks until replicas acknowledge; strong consistency, commit latency bound by cross-region RTT) versus asynchronous replication (commit returns immediately; replicas catch up; lag is a monitored, bounded quantity). FR1 requires both flows designed and explicitly selected per dataset.
3. Consistency axis: per-dataset guarantees. Strong (linearizable) for ownership records such as account balances; causal for feeds that depend on ordering; eventual with last-writer-wins or CRDT merge for counters and metrics. NFR4 explicitly permits eventual consistency for certain use cases, so the design must name which datasets get which guarantee.

Assumed baseline for this answer

Public figures anchor the category: DynamoDB has publicly reported serving 255.8 million requests per second at peak for Prime Day 2022 (Amazon, 2022); Google Spanner publicly describes synchronous Paxos replication across zones and regions with TrueTime-bounded clock uncertainty; Kafka MirrorMaker 2 is the open-source standard for cross-cluster topic replication and is deployed at LinkedIn, which publicly describes operating Kafka across multiple datacenters with MirrorMaker-style mirroring. Unless tied to one of those, every number in this answer is an explicit design assumption.

Assumed system: 3 regions (us-east, eu-west, ap-southeast), 2 billion replication events per day, 1 KB average payload, 3 writer regions for a multi-active subset and single-writer for the transactional core, and a five-times event peak. The architecture separates a control plane (topology, checkpoints, conflict policy) from a data plane (streams and storage), because the control plane must stay strongly consistent while the data plane is allowed to be eventually consistent with measured lag.