Redundancy and failover
How to keep a system up when parts fail: availability maths, active-passive and active-active, health checks, failover, RPO and RTO, and spreading copies across zones.
On this page
- The problem: everything fails eventually
- The idea: more than one of everything that matters
- Measuring it: availability and the nines
- Active-passive and active-active
- How failover actually works
- RPO and RTO: how much data, how much time
- Spreading copies apart: zones and regions
- Where this breaks
- When not to use it
- In the interview
- Quiz
The problem: everything fails eventually
A small online shop runs on one load balancer, two app servers and one database. One Tuesday at 03:10 the database's disk fails. Nothing is wrong with the code, the app servers are healthy, but every page that touches the database returns an error until someone wakes up, restores last night's backup to a new machine and points the app at it. The shop is down for four hours, and it has lost every order placed since the backup ran.
One database dies, and the whole app is down
- A single point of failure is any part whose failure takes the whole system down. One load balancer, one database, even one power cable: if it breaks, everything breaks.
The database was a single point of failure (SPOF): one part whose failure takes the whole system down. At scale, failure is not an accident but a rate. Disks, servers, network switches and power supplies fail a few percent a year each, so a fleet of 1,000 servers loses one somewhere every week or two. Software deploys, configuration changes and expired certificates fail far more often than hardware. The question is not whether parts fail, but whether users notice.
The idea: more than one of everything that matters
Redundancy means having more than one of each critical part, arranged so that any one of them can fail while the others carry on. Failover is the act of switching work from a failed part to a healthy one. Together they turn "the database died" from an outage into an alert.
Two of each part, and a standby database that takes over
- Redundancy means having more than one of each critical part, so any single one can fail without users noticing.
- In active-passive setups, a standby waits quietly and takes over when the main one fails. That switch is a failover, and it takes a few seconds. In active-active setups, all copies share the work all the time, so losing one just means the others carry a bit more.
Redundancy has three parts, and a design that skips any of them is not really redundant:
- Spare capacity: a second copy that can actually take the load.
- Detection: something notices the failure quickly and reliably.
- Switching: traffic and state move to the copy, automatically and safely.
Measuring it: availability and the nines
Availability is the fraction of time a system is working: uptime ÷ total time. It is usually quoted in "nines".
| Availability | Downtime per year | Downtime per 30 days | Typical of |
|---|---|---|---|
| 99% ("two nines") | 3.65 days | 7.2 hours | A single server with manual recovery |
| 99.9% | 8.8 hours | 43 minutes | A decent single-region service |
| 99.95% | 4.4 hours | 22 minutes | Many cloud services' SLAs |
| 99.99% | 53 minutes | 4.3 minutes | Redundant, multi-zone, automatic failover |
| 99.999% | 5.3 minutes | 26 seconds | Telecom-grade; very expensive |
Each extra nine allows ten times less downtime
- Availability is measured in “nines”. 99.9% sounds great but still allows almost 9 hours of downtime a year. 99.99% allows 52 minutes, and 99.999% just 5 minutes. Each extra nine costs a lot more.
Another way to see it: availability ≈ MTBF ÷ (MTBF + MTTR), where MTBF is the mean time between failures and MTTR the mean time to recover. A server that fails once a year and takes 8 hours to fix is at 99.9%. Make recovery automatic, taking 30 seconds, and the same unreliable server behind it gives about 99.9999% for that kind of failure. Shortening recovery is usually cheaper than preventing failure, and redundancy is mostly a way to make recovery fast.
Series and parallel: the two rules
Parts in a chain multiply; copies side by side multiply their failures
- In series (a request needs all of them): multiply the availabilities. LB 99.99% × app 99.9% × database 99.9% = 99.79%. Every extra dependency on the critical path lowers availability.
- In parallel (any one copy is enough): multiply the unavailabilities. Two copies at 99.9% are down together only 0.001 × 0.001 of the time: 99.9999%.
Worked example. The shop wants 99.95%. Its chain is LB → app → database.
| Design | LB | App tier | Database | Total |
|---|---|---|---|---|
| One of each, each 99.9% | 99.9% | 99.9% | 99.9% | 99.7% (26 h/year down) |
| Two LBs, three apps, primary + standby | 99.9999% | ~100% | 99.99% (failover ~60 s) | ≈ 99.99% |
The database dominates: even with a standby, each failover takes some time, so it is the weakest link. In practice the parallel numbers are optimistic, for two reasons covered below: copies do not fail independently, and failover itself can fail.
Active-passive and active-active
| Active-passive | Active-active | |
|---|---|---|
| How | One copy serves; a standby waits and is promoted on failure | All copies serve traffic all the time |
| Failover time | Seconds to minutes (detect, promote, redirect) | Near zero: the load balancer stops sending to the dead copy |
| Capacity | Standby is idle (you pay for it anyway) | All copies do useful work |
| Hard part | Detecting failure correctly; avoiding two primaries | Keeping state consistent when several copies accept writes |
| Use for | Databases with one writer, stateful leaders | Stateless app servers, caches, read replicas, multi-region reads |
Standbys come in temperatures: a hot standby is running and receiving every change (seconds to take over), a warm standby runs smaller and needs scaling up (minutes), and a cold standby is a backup plus a recipe for building a new server (hours).
Stateless app servers are easy to run active-active: keep sessions in a shared store such as Redis, not in server memory, and any server can handle any request. Databases are harder, because two copies that both accept writes can accept conflicting writes. Most designs run the database active-passive (one primary, replicas that can be promoted) and keep active-active for the stateless tiers. See replication for leader election and conflicts.
How failover actually works
Failover time is the sum of three delays: detect + decide + redirect.
1. Detect: health checks and heartbeats
A load balancer sends each server a health check every few seconds (for example GET /healthz every 5 s) and removes it after a few failures in a row (say 3), so detection takes 10–15 seconds. Two kinds of check matter:
- Liveness: "is the process alive?" If not, restart it.
- Readiness: "can it serve traffic right now?" If not, stop sending it requests, but don't kill it (it may be warming a cache or waiting for a dependency).
A shallow check (the process answers) misses a server that is up but broken. A deep check (it can reach the database) catches more, but can take every server out of rotation at once when the shared dependency blips, which turns a partial problem into a full outage. A good compromise: readiness checks the server's own health and critical local resources, and dependencies are handled with timeouts and circuit breakers.
2. Decide: who says the primary is dead?
For a database primary, a wrong decision is worse than a slow one. If the standby is promoted while the old primary is merely slow or cut off by the network, you get two primaries accepting writes: split brain. The protections:
- Quorum: a majority of independent observers (three or five nodes, often in different zones) must agree the primary is gone. Tools such as Patroni (with etcd), Orchestrator and cloud-managed databases do this.
- Fencing: before promoting, make sure the old primary cannot write: revoke its storage access, cut its power ("shoot the other node in the head", STONITH), or give every primary an epoch number and have storage reject writes with an old one.
3. Redirect: point clients at the new copy
| Method | Speed | Catch |
|---|---|---|
| Load balancer removes the dead server | Seconds | Only for servers behind an LB |
| Floating (virtual) IP moved to the standby | Seconds | Usually limited to one network or zone |
| DNS record updated | TTL + client caching: 30 s to minutes | Some clients ignore TTLs and cache for longer |
| Clients discover the primary (driver or proxy asks a coordinator) | Seconds | Needs smart clients or a proxy such as PgBouncer or ProxySQL |
Managed databases with a standby in another zone typically fail over in one to two minutes end to end, including DNS. Plan for that, and make clients reconnect and retry rather than crash.
RPO and RTO: how much data, how much time
Two numbers describe what a failure is allowed to cost, and you should agree them with the business per system:
- RPO (recovery point objective): how much recent data you may lose, measured in time. "RPO 5 minutes" means losing up to 5 minutes of writes is acceptable.
- RTO (recovery time objective): how long you may be down before service is restored.
RPO is decided by replication. With asynchronous replication the primary acknowledges a write before the standby has it, so a crash loses whatever had not yet been copied (usually under a second, but unbounded if the replica was lagging). With synchronous replication the primary waits for the standby to confirm, so RPO is zero, but every write pays the round trip, and if the standby is unavailable the primary must either stop accepting writes or fall back to asynchronous.
RTO is decided by how much is automated and how warm the standby is.
| Strategy | RPO | RTO | Cost |
|---|---|---|---|
| Nightly backups, restore by hand | Up to 24 hours | Hours | Lowest |
| Backups + continuous log shipping (point-in-time recovery) | Minutes | An hour or more | Low |
| Warm standby, async replication | Seconds | Minutes | Medium |
| Hot standby in another zone, sync replication, automatic failover | Zero | 1–2 minutes | High |
| Active-active in several regions | Near zero (with conflict handling) | Seconds | Highest |
Spreading copies apart: zones and regions
Copies in separate zones survive a power cut in one of them
- Copies only help if they fail independently. Two servers in the same rack share power and network. Clouds offer availability zones: separate buildings with their own power, so a fire or outage in one doesn’t take the other.
Copies only help if they fail independently. Two servers in the same rack share a power feed and a top-of-rack switch; two VMs may even share a physical host. Cloud providers offer three levels of separation:
| Level | Separate | Round trip between copies | Protects against |
|---|---|---|---|
| Different hosts / racks (placement groups) | Hardware, sometimes power | < 0.5 ms | A server or rack failing |
| Availability zones in one region | Buildings, power, cooling, networking | ~1–2 ms | A data centre fire, flood or power loss |
| Regions | Hundreds or thousands of km | 30–150 ms | A region-wide outage, a natural disaster |
The default for anything important is multi-AZ: app servers in at least two (preferably three) zones behind a load balancer, and a database with a synchronous standby in another zone. Zones are close enough that synchronous replication costs only a millisecond or two. Multi-region is a much bigger step: synchronous replication across 100 ms is usually too slow, so you accept asynchronous replication (non-zero RPO), routing users by region, and the complexity of deciding which region owns each piece of data. Reach for it when the business genuinely needs to survive a whole region failing, or to serve users far away with low latency.
Where this breaks
The single points that are easy to forget
- Don’t forget the less obvious single points: the load balancer itself, DNS, a shared config server, or one person who knows how to deploy. Then test it on purpose: teams deliberately break things in practice drills to prove failover works.
The hidden single points are often not servers at all: the one person who knows how to deploy, the one laptop with the signing key, the one region holding your build artefacts, the payment provider with no fallback.
When not to use it
Each extra nine costs roughly an order of magnitude more effort, and redundancy adds its own failure modes (split brain, failover bugs, replication lag). Skip or limit it when:
- The cost of downtime is low: an internal dashboard, a batch report, a prototype. A single server with good backups (and a known restore time) may be the right answer.
- The dependency chain caps you anyway: if you depend on a third party at 99.9%, spending months taking your own service from 99.99% to 99.999% buys little.
- Simpler recovery is enough: a stateless service on an autoscaling group that replaces dead instances in two minutes may already meet the target without hand-built failover.
In the interview
A checkout request goes through: a load balancer pair (99.99%), an app tier of 4 servers across 2 zones (treat it as 99.99%), a payments service you call synchronously (99.9%, a third party) and a Postgres primary with a standby (99.95% including failover time). What is checkout's availability, how many minutes of downtime a month is that, and what would you change first?
Compare with our answer
- Series: 0.9999 × 0.9999 × 0.999 × 0.9995 ≈ 0.9983, so about 99.83%, roughly 73 minutes a month (43,200 minutes × 0.0017).
- The third-party payments call dominates. You cannot make it more available, but you can take it off the critical path: accept the order, store it, and charge asynchronously with retries (and an idempotency key), or add a second payment provider as a fallback.
- Next is the database: a faster failover (smart clients or a proxy instead of DNS) moves it toward 99.99%.
- Making the already-good LB or app tier better barely changes the total.
A strong answer mentions: multiplying availabilities in series · converting to minutes per month · finding the weakest link rather than polishing strong ones · taking a dependency off the synchronous path · a fallback provider or asynchronous retry with idempotency.
A team runs two app servers in active-active, each at about 75% CPU at peak, in one availability zone. Each server's /health endpoint runs SELECT 1 against the database, and the load balancer removes a server after 2 failed checks. The database has an asynchronous replica in the same zone that they plan to promote by hand "if needed". List every problem.
Compare with our answer
- No N+1: at 75% each, losing one server needs 150% on the survivor, so it fails too.
- One zone: a zone outage takes the servers, the database and the replica together.
- Deep health check: a 10-second database blip fails every server's check at once, so the load balancer removes all of them and the site goes fully down, even for pages that don't need the database. Use a readiness check of the server itself, and handle the database with timeouts and circuit breakers.
- Manual promotion: RTO is "however long it takes someone to wake up", and an untested procedure. Automate it with quorum and fencing.
- Async replica: RPO is non-zero, and replication lag is not monitored.
- No backups mentioned: a replica does not protect against a bad
DELETE.
A strong answer mentions: capacity for one copy failing · spreading across zones · the danger of dependency checks in health endpoints · automated, fenced failover · RPO from async replication · backups as separate from replicas.
- SPOF: any one part whose failure stops everything, including people, DNS, config and certificates.
- Nines: 99.9% ≈ 8.8 h/year (43 min/month); 99.99% ≈ 53 min/year; 99.999% ≈ 5 min/year.
- Series multiply availabilities (chains get worse). Parallel: 1 − (1 − A)ⁿ, only if failures are independent.
- Availability ≈ MTBF ÷ (MTBF + MTTR): fast automatic recovery is the cheap lever.
- Active-active for stateless tiers (N+1 capacity!). Active-passive for single-writer databases; hot, warm or cold standby.
- Failover = detect + decide + redirect: health checks (readiness vs liveness), quorum and fencing against split brain, LB / VIP / DNS / smart clients.
- RPO (data you may lose) comes from replication: sync = 0, async = lag. RTO (time down) comes from automation.
- Multi-AZ by default (~1–2 ms apart, sync replication OK); multi-region means async and much more complexity.
- Replicas are not backups. Correlated failures (deploys, shared dependencies) beat redundancy: stage rollouts. Test failover on purpose.
1. A request passes through a load balancer (99.99%), an app tier (99.95%) and a database (99.9%), each required. Roughly what is the end-to-end availability?
0.9999 × 0.9995 × 0.999 ≈ 0.9984. A chain is worse than its weakest link, since any part failing breaks the request. The "weakest link" answer is the most tempting, but it ignores the others' downtime.
2. Two app servers run active-active at 80% CPU each during peak. One crashes. What happens?
Redundancy needs spare capacity. Two servers at 80% hold 160% of one server's capacity. Plan N+1: with any one copy gone, the rest must still fit. Autoscaling usually takes minutes, far longer than the overload takes to topple the survivor.
3. A team says: "We replicate our database synchronously to a standby in another zone, so we don't need backups." What is the main flaw?
Replication protects against losing a machine, not against bad writes, which it faithfully copies. Point-in-time backups protect against those. Sync replication loses no acknowledged writes, and cross-zone sync costs only a millisecond or two.
4. A payments database has RPO = 0 and RTO = 2 minutes. Which setup meets both?
RPO 0 needs synchronous replication (no acknowledged write exists only on the primary), and a 2-minute RTO needs automatic promotion. Async loses the lag on a crash; last-write-wins silently drops writes, which is not RPO 0 for payments.
5. Each app server's health check queries the shared database. The database stalls for 20 seconds. What is the most likely result?
A deep check makes every server's health depend on one shared component, so they all fail at once. Readiness checks should reflect the server itself; dependency trouble is handled with timeouts, circuit breakers and fallbacks.
6. The primary database is cut off by a network partition but keeps running. The standby is promoted. Why must the old primary be fenced?
Without fencing (revoking access, powering off, or rejecting writes from an old epoch), both nodes accept writes: split brain, with two histories that cannot be merged cleanly.
7. Your service runs in three availability zones with automatic failover, yet its worst outage last year hit all three zones at once. What is the most likely cause?
Zones are built to fail independently for hardware, power and network. Software and configuration are shared by every copy, so a bad push defeats redundancy. Staged rollouts (canary, one zone, then the rest) with automatic rollback limit the blast radius.
Score 60% or more and the lesson is marked complete.