Interview patterns · Design 14

Multi-region systems

Why and how to run in several regions: active-passive versus active-active, routing users, replication lag and write conflicts, data residency, and failover drills.

AdvancedInterview priority: Medium30 min
On this page
  1. The problem: one region, users everywhere, and a bad Tuesday
  2. Signals that you are facing it
  3. The idea: decide who may write what, where
  4. The toolbox
  5. Worked example: adding Europe
  6. When not to use it
  7. Which case studies use it
  8. In the interview
  9. Quiz

The problem: one region, users everywhere, and a bad Tuesday

A collaboration app runs entirely in one cloud region in Virginia. It has two growing problems.

  • Latency. 40% of users are in Europe. Each request from Frankfurt to Virginia costs a round trip of roughly 90 ms before any work is done, and opening a document makes five sequential API calls: almost half a second of pure waiting.
  • Availability. The region has a serious outage once every year or two (a power event, a networking fault, a broken control plane). Multiple availability zones (separate data centres within one region) protect against one building failing, not against the region's shared services failing. During the last outage, the app was down for six hours.

Then a large German customer asks for its data to be stored in the EU. Adding a second region solves all three, but it makes everything that used to be simple, such as "read your latest write", "one copy of each record" and "fail over the database", hard again.

Signals that you are facing it

  • Users on several continents and a tight latency target.
  • An availability target above what one region gives (often stated as "survive a region outage" or 99.99%+).
  • Data residency: "EU customers' data must stay in the EU".
  • Disaster recovery requirements stated as RPO and RTO (below).
  • The interviewer asks "what happens if us-east-1 goes down?"

Define the two numbers early. RPO (recovery point objective) is how much recent data you can afford to lose in a disaster, measured in time. RTO (recovery time objective) is how long you can be down. "RPO 1 minute, RTO 15 minutes" leads to a very different design from "RPO 0, RTO 30 seconds".

The idea: decide who may write what, where

Running stateless servers in two regions is easy. The hard part is data: copies in two places drift apart by the replication lag, and if both places accept writes to the same record, they can disagree. Every multi-region design is an answer to one question: for each piece of data, which region is allowed to write it?

The toolbox

Active-passive

One region (active) serves all traffic. A second (passive or standby) receives the data by asynchronous replication and runs a scaled-down copy of the stack. On disaster, you promote the standby and move traffic.

Active-passive, and what a failover loses

Good: simple, no write conflicts, cheap if the standby is small. Catch: no latency gain for remote users; RPO equals the replication lag at the moment of failure (any writes not yet copied are lost); and a standby that never takes traffic tends not to work when you need it.

Active-active with a home region per user or record

Both regions serve traffic. Each user (or tenant, or document) has a home region that owns its writes; the other region holds a replica for local reads and for failover. A European user's writes go to Frankfurt, an American user's to Virginia. Each record has exactly one writer, so there are no conflicts, and most requests stay local.

Active-active, each user writing in a home region

Catch: data shared across homes (a document edited by users from both continents) still needs one owner, so someone pays the cross-region trip. Moving a user's home region is a migration, not a flag.

Active-active, any region writes anything

Every region accepts writes to every record, and changes replicate in all directions (multi-leader replication). Writes are always local and fast, but two regions can change the same record at the same time, a write conflict.

Two regions write the same record, and the home-region fix

Ways to resolve conflicts:

MethodHowCatch
Last-writer-wins (LWW)Keep the write with the later timestampSilently drops the other write; clock skew picks the "wrong" winner
CRDTsData types whose concurrent changes always merge (counters, sets, text sequences)Only for data that fits those types; can't enforce rules like "balance ≥ 0"
Application mergeKeep both versions and merge (or ask the user)Complex code; conflicts surface to users
Avoid themRoute each record's writes to one home regionCross-region latency for remote writers

DynamoDB global tables and Cassandra's multi-datacentre mode use LWW by default, which is fine for a user's settings and wrong for a bank balance.

Globally consistent databases

Spanner and CockroachDB run a consensus protocol (Paxos or Raft) across regions: a write commits when a majority of replicas, in different regions, have it. You get RPO 0 and no conflicts, at the cost of write latency: every write waits for at least one round trip to another region, tens to over a hundred milliseconds depending on placement. Placing the majority of replicas near the main writers (with a "home region" per table or row) brings it down.

Routing users to a region

MethodHowCatch
GeoDNS / latency-based DNSDNS answers with the nearest healthy region (e.g. Route 53 latency routing)Clients and resolvers cache answers past the TTL, so failover takes minutes for some
AnycastEvery region announces the same IP address; the internet's routing delivers to the nearestNeeds a provider (Cloudflare, AWS Global Accelerator, GCP's global load balancer)
Home-region redirectThe edge looks up the user's home region and forwards thereOne lookup per session; must be cached at the edge

Usually the answer combines them: anycast or DNS gets the user to a nearby edge; the edge routes requests for user data to that user's home region. Static content always comes from a CDN.

Data residency

Some data must stay in a jurisdiction. The EU's GDPR does not require storage in the EU, but it restricts transfers outside it unless safeguards apply, so many companies keep EU personal data in EU regions; some other countries' laws do require local storage of certain data (India's rules for payment data are a well-known example). Design for it with home regions per tenant: the tenant's records, backups and search indexes live only in its region. The global layer holds only what is allowed to be global, such as a routing table from tenant ID to region.

Worked example: adding Europe

The app has 5,000 writes a second, about 1 KB each, and users split 60% US, 40% EU.

  • Latency now: a Frankfurt user's five sequential calls to Virginia cost about 5 × 90 ms = 450 ms of round trips. With an EU home region, that drops to a few ms each.
  • Replication traffic: 5,000 × 1 KB = 5 MB/s per direction ≈ 430 GB a day. Cross-region transfer is billed per GB (roughly a couple of cents per GB between many cloud regions), so a few hundred dollars a month per direction, before compression.
  • RPO with async replication: if lag is typically 1 s, a sudden failover loses up to about 5,000 writes. If that is unacceptable for some data (payments), that data needs synchronous or consensus replication, and its writes pay a cross-region round trip.
  • Capacity for failover: with two active regions, each must be able to take all traffic when the other fails, so each runs below 50% of its capacity normally, or autoscaling must add capacity faster than your RTO. With three regions, each can run at about 60%: if one fails, the other two absorb its third.

When not to use it

  • One region with several availability zones meets your target. It survives most failures; a full regional outage is rare. Many large products run single-region with a cold or warm standby.
  • Latency is a CDN problem. If most bytes are static (images, video, scripts), a CDN gives remote users most of the speed-up without moving any data.
  • Writes need global strong consistency and low latency together. Physics forbids it: a write that every region must agree on costs at least one cross-region round trip.

Which case studies use it

  • Design WhatsApp: users connect to nearby regions; messages route to the recipient's region.
  • Design Google Docs: each document has one home server/region that orders edits, while viewers connect from anywhere.
  • Design a payment system: RPO 0 for the ledger, so synchronous or consensus replication, with residency for card data.
  • Design Zoom: media servers placed near participants; a meeting is anchored in one region.
  • Design Amazon checkout: cell-based regions and failover drills for an always-on checkout.

In the interview

Your turn: a global e-commerce site

A shop sells in the US, the EU and Japan. Product catalogue edits come from one team in the US; orders and carts belong to customers. Requirements: page loads under 200 ms everywhere, survive a region outage with at most a few seconds of lost orders, EU customer data stored in the EU. Design the regional layout.

Compare with our answer
  • Three regions, one per market; customers get a home region (EU customers in the EU for residency).
  • Catalogue: single writer (US), replicated asynchronously to all regions and served through a CDN and local caches. Seconds of lag are fine.
  • Carts and orders: written in the customer's home region. Replicate each region's orders to a standby in another region within the same jurisdiction where required (EU data to a second EU region), giving RPO of about the lag.
  • Inventory: one stock count per warehouse, owned by the warehouse's region; selling across regions reserves stock with a call to the owner, not a local guess (see contention).
  • Routing: anycast/DNS to the nearest edge, which forwards to the home region; headroom in each region for a neighbour's traffic.

A strong answer mentions: home regions per customer · single-writer catalogue replicated with a CDN · stated RPO from async lag · residency including backups · an owner for shared, contended data like stock · failover capacity and drills.

Your turn: spot the bug

A team runs two regions active-active with a multi-leader database using last-writer-wins. Wallet balances are stored as a balance column. After a network partition between the regions heals, finance finds that several customers spent the same money twice. What happened, and what would you change?

Compare with our answer

During the partition each region kept accepting writes. A customer spent their balance in region A and the same balance in region B; each region checked balance ≥ amount against its own copy. When replication resumed, LWW kept one of the two final balances and discarded the other update, so one purchase was never deducted.

A strong answer mentions:

  • Conflicting writes to the same record in two regions; LWW drops one silently.
  • A rule like "balance never below zero" needs a single decision point: route each wallet's writes to its home region (or use a consensus database).
  • Store money as a ledger of entries rather than an overwritable balance, so nothing is lost and reconciliation can detect problems.
  • During a partition, the non-home region must refuse or queue wallet writes (choose consistency over availability for this data; see consistency and CAP).
  • Monitor replication and alert on conflicts.
Multi-region on one page
  • Reasons: latency (cross-continent RTT ~100 ms), surviving a region outage, data residency.
  • State RPO (data you may lose) and RTO (time you may be down) first.
  • Active-passive: simple, no conflicts; RPO = replication lag; no latency gain; drill it or it won't work.
  • Active-active with home regions: one writer per user/record, local reads, no conflicts. The usual answer.
  • Any-region writes: conflicts. LWW drops data; CRDTs for mergeable types; avoid for money and limits.
  • Consensus databases (Spanner, CockroachDB): RPO 0, writes pay a cross-region round trip.
  • Routing: anycast or latency DNS to the edge, edge to home region; CDN for static content.
  • Residency covers backups, logs, caches, analytics, not just the main database.
  • Capacity: two regions run below 50%; fence the old primary; failover drills on a schedule.
Check what stuck

1. A company runs active-passive with asynchronous replication that usually lags 2 seconds. The primary region fails suddenly. What is the best estimate of data loss?

2. Users in the US and EU both edit shared project boards. The team wants local-speed reads and no lost edits. What fits best?

3. An EU-only deployment must keep customer data in the EU. Which is the most likely accidental violation?

4. Two active regions each run at 70% CPU at peak. One region fails. What happens?

5. A team uses CockroachDB with replicas in Virginia, Frankfurt and Tokyo. Writes take 120 ms. Why?

6. After a failover drill, the old primary rejoins and both regions accept writes for several minutes. What prevents this?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.