Staying up · Chapter 14

Redundancy and failover

How to keep a system up when parts fail: availability maths, active-passive and active-active, health checks, failover, RPO and RTO, and spreading copies across zones.

IntermediateInterview priority: High40 min
On this page
  1. The problem: everything fails eventually
  2. The idea: more than one of everything that matters
  3. Measuring it: availability and the nines
  4. Active-passive and active-active
  5. How failover actually works
  6. RPO and RTO: how much data, how much time
  7. Spreading copies apart: zones and regions
  8. Where this breaks
  9. When not to use it
  10. In the interview
  11. Quiz

The problem: everything fails eventually

A small online shop runs on one load balancer, two app servers and one database. One Tuesday at 03:10 the database's disk fails. Nothing is wrong with the code, the app servers are healthy, but every page that touches the database returns an error until someone wakes up, restores last night's backup to a new machine and points the app at it. The shop is down for four hours, and it has lost every order placed since the backup ran.

One database dies, and the whole app is down

  1. A single point of failure is any part whose failure takes the whole system down. One load balancer, one database, even one power cable: if it breaks, everything breaks.

The database was a single point of failure (SPOF): one part whose failure takes the whole system down. At scale, failure is not an accident but a rate. Disks, servers, network switches and power supplies fail a few percent a year each, so a fleet of 1,000 servers loses one somewhere every week or two. Software deploys, configuration changes and expired certificates fail far more often than hardware. The question is not whether parts fail, but whether users notice.

The idea: more than one of everything that matters

Redundancy means having more than one of each critical part, arranged so that any one of them can fail while the others carry on. Failover is the act of switching work from a failed part to a healthy one. Together they turn "the database died" from an outage into an alert.

Two of each part, and a standby database that takes over

  1. Redundancy means having more than one of each critical part, so any single one can fail without users noticing.
  2. In active-passive setups, a standby waits quietly and takes over when the main one fails. That switch is a failover, and it takes a few seconds. In active-active setups, all copies share the work all the time, so losing one just means the others carry a bit more.

Redundancy has three parts, and a design that skips any of them is not really redundant:

  1. Spare capacity: a second copy that can actually take the load.
  2. Detection: something notices the failure quickly and reliably.
  3. Switching: traffic and state move to the copy, automatically and safely.

Measuring it: availability and the nines

Availability is the fraction of time a system is working: uptime ÷ total time. It is usually quoted in "nines".

AvailabilityDowntime per yearDowntime per 30 daysTypical of
99% ("two nines")3.65 days7.2 hoursA single server with manual recovery
99.9%8.8 hours43 minutesA decent single-region service
99.95%4.4 hours22 minutesMany cloud services' SLAs
99.99%53 minutes4.3 minutesRedundant, multi-zone, automatic failover
99.999%5.3 minutes26 secondsTelecom-grade; very expensive

Each extra nine allows ten times less downtime

  1. Availability is measured in “nines”. 99.9% sounds great but still allows almost 9 hours of downtime a year. 99.99% allows 52 minutes, and 99.999% just 5 minutes. Each extra nine costs a lot more.

Another way to see it: availability ≈ MTBF ÷ (MTBF + MTTR), where MTBF is the mean time between failures and MTTR the mean time to recover. A server that fails once a year and takes 8 hours to fix is at 99.9%. Make recovery automatic, taking 30 seconds, and the same unreliable server behind it gives about 99.9999% for that kind of failure. Shortening recovery is usually cheaper than preventing failure, and redundancy is mostly a way to make recovery fast.

Series and parallel: the two rules

Parts in a chain multiply; copies side by side multiply their failures

  • In series (a request needs all of them): multiply the availabilities. LB 99.99% × app 99.9% × database 99.9% = 99.79%. Every extra dependency on the critical path lowers availability.
  • In parallel (any one copy is enough): multiply the unavailabilities. Two copies at 99.9% are down together only 0.001 × 0.001 of the time: 99.9999%.

Worked example. The shop wants 99.95%. Its chain is LB → app → database.

DesignLBApp tierDatabaseTotal
One of each, each 99.9%99.9%99.9%99.9%99.7% (26 h/year down)
Two LBs, three apps, primary + standby99.9999%~100%99.99% (failover ~60 s)≈ 99.99%

The database dominates: even with a standby, each failover takes some time, so it is the weakest link. In practice the parallel numbers are optimistic, for two reasons covered below: copies do not fail independently, and failover itself can fail.

Active-passive and active-active

Active-passiveActive-active
HowOne copy serves; a standby waits and is promoted on failureAll copies serve traffic all the time
Failover timeSeconds to minutes (detect, promote, redirect)Near zero: the load balancer stops sending to the dead copy
CapacityStandby is idle (you pay for it anyway)All copies do useful work
Hard partDetecting failure correctly; avoiding two primariesKeeping state consistent when several copies accept writes
Use forDatabases with one writer, stateful leadersStateless app servers, caches, read replicas, multi-region reads

Standbys come in temperatures: a hot standby is running and receiving every change (seconds to take over), a warm standby runs smaller and needs scaling up (minutes), and a cold standby is a backup plus a recipe for building a new server (hours).

Stateless app servers are easy to run active-active: keep sessions in a shared store such as Redis, not in server memory, and any server can handle any request. Databases are harder, because two copies that both accept writes can accept conflicting writes. Most designs run the database active-passive (one primary, replicas that can be promoted) and keep active-active for the stateless tiers. See replication for leader election and conflicts.

How failover actually works

Failover time is the sum of three delays: detect + decide + redirect.

1. Detect: health checks and heartbeats

A load balancer sends each server a health check every few seconds (for example GET /healthz every 5 s) and removes it after a few failures in a row (say 3), so detection takes 10–15 seconds. Two kinds of check matter:

  • Liveness: "is the process alive?" If not, restart it.
  • Readiness: "can it serve traffic right now?" If not, stop sending it requests, but don't kill it (it may be warming a cache or waiting for a dependency).

A shallow check (the process answers) misses a server that is up but broken. A deep check (it can reach the database) catches more, but can take every server out of rotation at once when the shared dependency blips, which turns a partial problem into a full outage. A good compromise: readiness checks the server's own health and critical local resources, and dependencies are handled with timeouts and circuit breakers.

2. Decide: who says the primary is dead?

For a database primary, a wrong decision is worse than a slow one. If the standby is promoted while the old primary is merely slow or cut off by the network, you get two primaries accepting writes: split brain. The protections:

  • Quorum: a majority of independent observers (three or five nodes, often in different zones) must agree the primary is gone. Tools such as Patroni (with etcd), Orchestrator and cloud-managed databases do this.
  • Fencing: before promoting, make sure the old primary cannot write: revoke its storage access, cut its power ("shoot the other node in the head", STONITH), or give every primary an epoch number and have storage reject writes with an old one.

3. Redirect: point clients at the new copy

MethodSpeedCatch
Load balancer removes the dead serverSecondsOnly for servers behind an LB
Floating (virtual) IP moved to the standbySecondsUsually limited to one network or zone
DNS record updatedTTL + client caching: 30 s to minutesSome clients ignore TTLs and cache for longer
Clients discover the primary (driver or proxy asks a coordinator)SecondsNeeds smart clients or a proxy such as PgBouncer or ProxySQL

Managed databases with a standby in another zone typically fail over in one to two minutes end to end, including DNS. Plan for that, and make clients reconnect and retry rather than crash.

RPO and RTO: how much data, how much time

Two numbers describe what a failure is allowed to cost, and you should agree them with the business per system:

  • RPO (recovery point objective): how much recent data you may lose, measured in time. "RPO 5 minutes" means losing up to 5 minutes of writes is acceptable.
  • RTO (recovery time objective): how long you may be down before service is restored.

RPO is decided by replication. With asynchronous replication the primary acknowledges a write before the standby has it, so a crash loses whatever had not yet been copied (usually under a second, but unbounded if the replica was lagging). With synchronous replication the primary waits for the standby to confirm, so RPO is zero, but every write pays the round trip, and if the standby is unavailable the primary must either stop accepting writes or fall back to asynchronous.

RTO is decided by how much is automated and how warm the standby is.

StrategyRPORTOCost
Nightly backups, restore by handUp to 24 hoursHoursLowest
Backups + continuous log shipping (point-in-time recovery)MinutesAn hour or moreLow
Warm standby, async replicationSecondsMinutesMedium
Hot standby in another zone, sync replication, automatic failoverZero1–2 minutesHigh
Active-active in several regionsNear zero (with conflict handling)SecondsHighest

Spreading copies apart: zones and regions

Copies in separate zones survive a power cut in one of them

  1. Copies only help if they fail independently. Two servers in the same rack share power and network. Clouds offer availability zones: separate buildings with their own power, so a fire or outage in one doesn’t take the other.

Copies only help if they fail independently. Two servers in the same rack share a power feed and a top-of-rack switch; two VMs may even share a physical host. Cloud providers offer three levels of separation:

LevelSeparateRound trip between copiesProtects against
Different hosts / racks (placement groups)Hardware, sometimes power< 0.5 msA server or rack failing
Availability zones in one regionBuildings, power, cooling, networking~1–2 msA data centre fire, flood or power loss
RegionsHundreds or thousands of km30–150 msA region-wide outage, a natural disaster

The default for anything important is multi-AZ: app servers in at least two (preferably three) zones behind a load balancer, and a database with a synchronous standby in another zone. Zones are close enough that synchronous replication costs only a millisecond or two. Multi-region is a much bigger step: synchronous replication across 100 ms is usually too slow, so you accept asynchronous replication (non-zero RPO), routing users by region, and the complexity of deciding which region owns each piece of data. Reach for it when the business genuinely needs to survive a whole region failing, or to serve users far away with low latency.

Where this breaks

The single points that are easy to forget

  1. Don’t forget the less obvious single points: the load balancer itself, DNS, a shared config server, or one person who knows how to deploy. Then test it on purpose: teams deliberately break things in practice drills to prove failover works.

The hidden single points are often not servers at all: the one person who knows how to deploy, the one laptop with the signing key, the one region holding your build artefacts, the payment provider with no fallback.

When not to use it

Each extra nine costs roughly an order of magnitude more effort, and redundancy adds its own failure modes (split brain, failover bugs, replication lag). Skip or limit it when:

  • The cost of downtime is low: an internal dashboard, a batch report, a prototype. A single server with good backups (and a known restore time) may be the right answer.
  • The dependency chain caps you anyway: if you depend on a third party at 99.9%, spending months taking your own service from 99.99% to 99.999% buys little.
  • Simpler recovery is enough: a stateless service on an autoscaling group that replaces dead instances in two minutes may already meet the target without hand-built failover.

In the interview

Your turn: work out the availability

A checkout request goes through: a load balancer pair (99.99%), an app tier of 4 servers across 2 zones (treat it as 99.99%), a payments service you call synchronously (99.9%, a third party) and a Postgres primary with a standby (99.95% including failover time). What is checkout's availability, how many minutes of downtime a month is that, and what would you change first?

Compare with our answer
  • Series: 0.9999 × 0.9999 × 0.999 × 0.9995 ≈ 0.9983, so about 99.83%, roughly 73 minutes a month (43,200 minutes × 0.0017).
  • The third-party payments call dominates. You cannot make it more available, but you can take it off the critical path: accept the order, store it, and charge asynchronously with retries (and an idempotency key), or add a second payment provider as a fallback.
  • Next is the database: a faster failover (smart clients or a proxy instead of DNS) moves it toward 99.99%.
  • Making the already-good LB or app tier better barely changes the total.

A strong answer mentions: multiplying availabilities in series · converting to minutes per month · finding the weakest link rather than polishing strong ones · taking a dependency off the synchronous path · a fallback provider or asynchronous retry with idempotency.

Your turn: spot the flaw

A team runs two app servers in active-active, each at about 75% CPU at peak, in one availability zone. Each server's /health endpoint runs SELECT 1 against the database, and the load balancer removes a server after 2 failed checks. The database has an asynchronous replica in the same zone that they plan to promote by hand "if needed". List every problem.

Compare with our answer
  • No N+1: at 75% each, losing one server needs 150% on the survivor, so it fails too.
  • One zone: a zone outage takes the servers, the database and the replica together.
  • Deep health check: a 10-second database blip fails every server's check at once, so the load balancer removes all of them and the site goes fully down, even for pages that don't need the database. Use a readiness check of the server itself, and handle the database with timeouts and circuit breakers.
  • Manual promotion: RTO is "however long it takes someone to wake up", and an untested procedure. Automate it with quorum and fencing.
  • Async replica: RPO is non-zero, and replication lag is not monitored.
  • No backups mentioned: a replica does not protect against a bad DELETE.

A strong answer mentions: capacity for one copy failing · spreading across zones · the danger of dependency checks in health endpoints · automated, fenced failover · RPO from async replication · backups as separate from replicas.

Redundancy on one page
  • SPOF: any one part whose failure stops everything, including people, DNS, config and certificates.
  • Nines: 99.9% ≈ 8.8 h/year (43 min/month); 99.99% ≈ 53 min/year; 99.999% ≈ 5 min/year.
  • Series multiply availabilities (chains get worse). Parallel: 1 − (1 − A)ⁿ, only if failures are independent.
  • Availability ≈ MTBF ÷ (MTBF + MTTR): fast automatic recovery is the cheap lever.
  • Active-active for stateless tiers (N+1 capacity!). Active-passive for single-writer databases; hot, warm or cold standby.
  • Failover = detect + decide + redirect: health checks (readiness vs liveness), quorum and fencing against split brain, LB / VIP / DNS / smart clients.
  • RPO (data you may lose) comes from replication: sync = 0, async = lag. RTO (time down) comes from automation.
  • Multi-AZ by default (~1–2 ms apart, sync replication OK); multi-region means async and much more complexity.
  • Replicas are not backups. Correlated failures (deploys, shared dependencies) beat redundancy: stage rollouts. Test failover on purpose.
Interactive labPractise it: survive server, database and zone failures on a budgetOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. A request passes through a load balancer (99.99%), an app tier (99.95%) and a database (99.9%), each required. Roughly what is the end-to-end availability?

2. Two app servers run active-active at 80% CPU each during peak. One crashes. What happens?

3. A team says: "We replicate our database synchronously to a standby in another zone, so we don't need backups." What is the main flaw?

4. A payments database has RPO = 0 and RTO = 2 minutes. Which setup meets both?

5. Each app server's health check queries the shared database. The database stalls for 20 seconds. What is the most likely result?

6. The primary database is cut off by a network partition but keeps running. The standby is promoted. Why must the old primary be fenced?

7. Your service runs in three availability zones with automatic failover, yet its worst outage last year hit all three zones at once. What is the most likely cause?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.