Staying up · Chapter 15

Timeouts, retries and circuit breakers

How one slow service takes down the services that call it, and the defences: timeouts, retries with backoff and jitter, retry budgets, circuit breakers, bulkheads and load shedding.

IntermediateInterview priority: High50 min
On this page
  1. The problem: one slow service takes down another
  2. The idea: contain the failure at the boundary
  3. Timeouts: never wait forever
  4. Retries: a second try, carefully
  5. Circuit breakers: stop trying for a while
  6. Bulkheads: one dependency cannot take everything
  7. Load shedding: refuse work you cannot finish
  8. Fallbacks and graceful degradation
  9. Choosing the right defence
  10. Where this breaks
  11. When not to use them
  12. In the interview
  13. Quiz

The problem: one slow service takes down another

A product page calls a recommendations service to show "you might also like". The page server has 50 worker threads and serves 100 pages a second; each page normally waits 20 ms for recommendations. Then recommendations has a bad deploy and starts taking 30 seconds to answer.

Nothing has failed outright. Recommendations is still accepting connections; it is just slow. But every page request now holds a worker for 30 seconds. Little's law says the number of requests in flight equals arrival rate × time in the system: 100 pages/s × 30 s = 3,000 workers needed, and there are 50. Within half a second every worker is stuck waiting, new requests queue, and the product page stops loading, including pages that would have been fine without recommendations.

A slow dependency ties up every worker, and the caller goes down too

  1. Services call other services. Checkout calls payments; product pages call recommendations. If the service you call gets slow, every request waiting on it piles up behind it.
  2. Soon the caller runs out of workers too, and pages that don’t even need recommendations stop loading. One slow service has knocked over another. That is a cascading failure.

This is a cascading failure: a problem in one service spreads up the call chain to everything that depends on it. A slow dependency is worse than a dead one, because a dead one fails fast (connection refused in a millisecond) while a slow one holds resources until something gives up. The defences in this lesson are all ways of giving up early, on purpose, and cheaply.

The idea: contain the failure at the boundary

Every call to another service is a place where a failure can cross from one system to another. Resilience patterns put a barrier at that boundary:

DefenceQuestion it answersProtects
TimeoutHow long will I wait?The caller's workers
Retry with backoffIs it worth trying again, and when?Users from one-off glitches
Retry budgetHow much extra load may retries add?The struggling dependency
Circuit breakerShould I even try right now?Both sides, during a sustained failure
BulkheadHow much of my capacity can one dependency use?Everything else the caller does
Load sheddingWhich requests do I refuse when I'm overloaded?The server itself
FallbackWhat do I show instead?The user experience

Timeouts: never wait forever

With a timeout, the page gives up and loads without recommendations

  1. Defence number one is a timeout: never wait forever. If recommendations don’t answer in 200 milliseconds, give up and move on.

Many HTTP clients and database drivers default to no timeout or a very long one (minutes). Always set one explicitly, and set the right kinds:

  • Connect timeout: how long to wait to establish a connection. Short (100–500 ms in one data centre), since a healthy server accepts in about a millisecond.
  • Request (read) timeout: how long to wait for the answer. Base it on the dependency's real latency: a little above its p99.9 in normal operation. If recommendations answers in 20 ms at p50 and 120 ms at p99.9, a 200–300 ms timeout is reasonable.

Deadlines, not just timeouts

Timeouts per hop add up. If the page has a 1-second budget and calls A (timeout 800 ms), which calls B (timeout 800 ms), B may keep working long after the user has given up. Pass a deadline down the chain instead: "this request must finish by 12:00:00.950". Each service checks the time remaining, uses that as its own timeout, and refuses work whose deadline has already passed. gRPC does this natively (the deadline travels with the call); with HTTP you can carry it in a header.

Worked example. The product page has an 800 ms budget. It calls the catalogue (required, p99.9 = 150 ms) and recommendations (optional, p99.9 = 120 ms) in parallel. Give the catalogue 400 ms, which leaves room for one retry within the budget, and recommendations a hard 200 ms with no retry, because the page is still useful without it. The worst case is then about 400 + 400 (one catalogue retry) = 800 ms, within budget, and a slow recommendations service can never cost more than 200 ms.

Retries: a second try, carefully

Many failures are brief: a dropped packet, a server restarting during a deploy, a leader election. A quick second try fixes them, so retries are worth having. They are also dangerous, because they add load exactly when a service is struggling.

Retries can storm a struggling service; backoff with jitter spreads them out

  1. Number two is the retry: a quick second try fixes many one-off glitches. But if a struggling service gets retried by thousands of callers at once, the extra traffic finishes it off: a retry storm. So retries wait longer each time, with a little randomness: exponential backoff with jitter.

Rules for safe retries:

  1. Only retry what is safe to repeat. Reads are fine. A POST /payments that timed out may have succeeded; retrying it can charge twice unless it carries an idempotency key that the server uses to recognise the repeat (see idempotency for how).
  2. Only retry errors that might go away: timeouts, connection resets, 503, 429 (after Retry-After). Never retry 400, 401, 403 or 404: they will fail the same way every time.
  3. Wait longer each time (exponential backoff): 100 ms, 200 ms, 400 ms… up to a cap.
  4. Add randomness (jitter). Without it, a thousand clients that failed at the same instant all retry at the same instant, again and again, in synchronised waves. With "full jitter" each client sleeps a random time between 0 and the backoff ceiling, which spreads the retries evenly.
  5. Limit attempts: two or three tries in total is usually enough for synchronous calls.
import java.util.concurrent.ThreadLocalRandom;
import java.util.function.Supplier;

public final class Backoff {
    /** Failures worth retrying: timeouts, 503s, connection resets. */
    public static class RetryableException extends RuntimeException {
        public RetryableException(String msg) { super(msg); }
    }

    public static <T> T withRetries(Supplier<T> call, int maxAttempts, long baseMs, long capMs)
            throws InterruptedException {
        for (int attempt = 0; ; attempt++) {
            try {
                return call.get();
            } catch (RetryableException e) {
                if (attempt == maxAttempts - 1) throw e;                 // out of attempts
                long ceiling = Math.min(capMs, baseMs << attempt);       // 100, 200, 400 ... capped
                Thread.sleep(ThreadLocalRandom.current().nextLong(ceiling + 1));  // full jitter
            }
        }
    }
}

String body = Backoff.withRetries(() -> catalogue.get(productId), 3, 100, 2_000);

Retries multiply across layers

Retries at every layer multiply; retry at one layer, within a budget

The worst retry bugs come from good intentions at every layer. If the app, the API and the orders service each try 3 times, one click becomes up to 27 calls to the database, and a database that was 10% overloaded is now hundreds of percent overloaded. Two fixes:

  • Retry at one layer, usually the one nearest the failing dependency, and have the others pass errors up quickly.
  • Use a retry budget: allow retries only while they are a small fraction of traffic, for example "retries may add at most 10% on top of normal requests". When most calls are failing, the budget runs out and the client stops retrying, so load on the dependency stays at about 1.1× instead of 3×. gRPC's retry throttling and Finagle's retry budgets work this way.

Circuit breakers: stop trying for a while

Timeouts and retries treat each call on its own. When a dependency is clearly down, it is pointless (and harmful) for every request to wait out a timeout and retry. A circuit breaker watches the results of recent calls and, when too many fail, "trips" so that calls fail immediately without touching the dependency.

The breaker opens on failures, then half-opens to test, then closes

  1. Number three is the circuit breaker. Like the one in your home, it trips when too many calls fail. While it’s open, calls fail instantly without even trying, which gives the struggling service room to recover.
  2. After a pause it goes half-open and lets one test call through. If that works, it closes and traffic flows again. If not, it stays open a while longer.

It has three states:

StateWhat happens to callsMoves to
Closed (normal)Calls go through; results are recorded in a sliding window (e.g. the last 20 calls)Open when the failure rate passes a threshold (e.g. 50%), once there are enough calls to judge
OpenCalls fail at once (or use the fallback), costing microsecondsHalf-open after a pause (e.g. 30 s)
Half-openA few test calls go throughClosed if they succeed; back to Open if they fail

The breaker helps both sides: the caller stops wasting workers on calls that will fail, and the struggling service gets a quiet period in which it can recover instead of being hammered. Count slow calls as failures too (a call over its timeout is a failure), and do not count errors that are the caller's fault, such as 400s, or one buggy client can trip the breaker for everyone.

Here is a complete count-based breaker. Production libraries add metrics, time-based windows and configurable probe counts, but the logic is the same.

import java.util.ArrayDeque;
import java.util.Deque;
import java.util.function.Supplier;

public final class CircuitBreaker {
    public enum State { CLOSED, OPEN, HALF_OPEN }

    private final int windowSize;            // judge the last N calls
    private final double failureThreshold;   // e.g. 0.5 = trip at 50% failures
    private final long openNanos;            // how long to stay open
    private final Deque<Boolean> window = new ArrayDeque<>();
    private State state = State.CLOSED;
    private long openedAt;
    private boolean probeInFlight;

    public CircuitBreaker(int windowSize, double failureThreshold, long openMillis) {
        this.windowSize = windowSize;
        this.failureThreshold = failureThreshold;
        this.openNanos = openMillis * 1_000_000L;
    }

    public synchronized State state() { return state; }

    private synchronized boolean tryAcquire() {
        if (state == State.OPEN) {
            if (System.nanoTime() - openedAt < openNanos) return false;   // fail fast
            state = State.HALF_OPEN;                                       // pause over: probe
        }
        if (state == State.HALF_OPEN) {
            if (probeInFlight) return false;      // only one test call at a time
            probeInFlight = true;
        }
        return true;
    }

    private synchronized void record(boolean success) {
        if (state == State.HALF_OPEN) {
            probeInFlight = false;
            if (success) { state = State.CLOSED; window.clear(); }
            else trip();
            return;
        }
        if (state == State.OPEN) return;          // late result from before the trip
        window.addLast(success);
        if (window.size() > windowSize) window.removeFirst();
        long failures = window.stream().filter(ok -> !ok).count();
        if (window.size() == windowSize && failures >= failureThreshold * windowSize) trip();
    }

    private void trip() {
        state = State.OPEN;
        openedAt = System.nanoTime();
        window.clear();
    }

    public <T> T call(Supplier<T> action, Supplier<T> fallback) {
        if (!tryAcquire()) return fallback.get();
        try {
            T result = action.get();
            record(true);
            return result;
        } catch (RuntimeException e) {
            record(false);
            return fallback.get();
        }
    }
}

CircuitBreaker recsBreaker = new CircuitBreaker(20, 0.5, 30_000);
List<Product> recs = recsBreaker.call(() -> recsClient.forUser(userId), () -> popularThisWeek());

In practice you use a library: Resilience4j in Java (Netflix's Hystrix, which popularised the pattern, has been in maintenance mode since 2018), Polly in .NET, pybreaker in Python, or the service mesh: Envoy and Istio provide "outlier detection", which ejects individual misbehaving hosts from the load-balancing pool, plus limits on connections and pending requests.

Bulkheads: one dependency cannot take everything

A ship is divided into watertight compartments (bulkheads) so one hole does not sink it. In software, a bulkhead gives each dependency its own limited pool of resources: say 10 concurrent calls for recommendations, 30 for the catalogue, 20 for payments, implemented as separate thread pools, semaphores or connection pools. When recommendations hangs, it can use up its own 10 slots and no more; the other 40 workers keep serving pages. Without bulkheads, a timeout only limits how long each call waits, not how many workers are waiting at once.

Load shedding: refuse work you cannot finish

Everything so far protects a caller. A server also needs to protect itself from more traffic than it can handle. Past full capacity, queues grow, every request waits longer, clients time out and retry, and the server ends up doing work for requests whose callers have already given up. Throughput can fall to near zero while the server is at 100% CPU.

Load shedding means rejecting excess requests early and cheaply (503 with Retry-After) so the ones you accept finish quickly:

  • Cap concurrency or queue length, and reject beyond it. Adaptive limiters (such as Netflix's concurrency-limits library) lower the cap automatically when latency rises.
  • Drop requests whose deadline has passed instead of processing them for nobody.
  • Shed by priority: checkout before recommendations, user-facing before batch, so the requests that matter most survive.

Rate limiting is the per-client version of the same idea; load shedding is about total capacity regardless of who is asking.

Fallbacks and graceful degradation

A sensible fallback keeps the page useful

  1. Finally, plan a fallback: something sensible to show when a dependency is unavailable, like “popular items” instead of personal recommendations. A slightly worse page beats an error page.

When a call fails, fast, the question is what to show. Good fallbacks, from best to most basic:

  • A cached or slightly stale answer: yesterday's recommendations, the last known exchange rate.
  • A generic answer: "popular this week" instead of personal recommendations.
  • Leave the feature out: the page without a reviews section.
  • A clear, fast error for something essential, such as "payments are temporarily unavailable, your basket is saved".

Decide in advance which features are critical (the page is useless without them) and which are optional (the page degrades). Only optional dependencies get silent fallbacks; for critical ones, fail fast and honestly.

Choosing the right defence

DefenceGood atWeak atUse when
TimeoutBounding the wait; cheapDoesn't reduce load on the dependencyAlways, on every network call
Retry + backoff + jitterHiding brief glitchesAdds load; unsafe for non-idempotent callsTransient errors on idempotent calls
Retry budgetStopping retry stormsSome recoverable calls go unretried during outagesAny client that retries at scale
Circuit breakerFast failure during sustained outages; recovery roomNeeds tuning; can flap or trip on the wrong errorsRemote dependencies that can be down for seconds or more
BulkheadIsolating one dependency's slownessUnused reserved capacityCallers with several dependencies of different importance
Load sheddingKeeping a server useful past capacitySome users get errorsEvery server that can receive more than it can handle
FallbackKeeping the user experience intactUntested fallback paths can fail tooOptional features

Where this breaks

When not to use them

  • Don't retry non-idempotent operations without an idempotency key.
  • Don't add a circuit breaker around a dependency you cannot work without and that has no failure mode except "slow", such as your own primary database, unless you have a meaningful fast-failure response; a timeout plus load shedding is often enough.
  • Don't retry inside asynchronous consumers with sleeps. A queue consumer should hand failed messages back to the queue (with a delay or dead-letter queue) instead of blocking a worker.
  • Don't wrap in-process calls: these patterns are for network boundaries.

In the interview

Your turn: checkout calls payments

A checkout service calls a third-party payment provider synchronously. Normally it answers in 300 ms (p99 1.2 s). Twice a month it slows to 20 s for a few minutes. Checkout has 200 workers and handles 50 checkouts a second. Design the timeouts, retries, breaker and fallback. What happens to a payment that timed out?

Compare with our answer
  • Capacity check: at 20 s per call, 50/s × 20 s = 1,000 workers needed, so without protection checkout falls over within 4 seconds. A bulkhead (say 100 concurrent payment calls) keeps the rest of checkout alive.
  • Timeout: around 2–3 s (above p99, below user patience).
  • Retries: a timed-out payment may have succeeded. Send an idempotency key with every charge so a retry cannot charge twice; retry at most once with jittered backoff, and only within a budget.
  • Breaker: trip when most recent calls fail or are slow; while open, fail fast.
  • Fallback: there is no fake payment. Options: accept the order as "payment pending", queue the charge and retry asynchronously, then email the customer; or fail with a clear message and keep the basket. A second payment provider is the strongest fallback.
  • Reconcile: an unknown-outcome payment is resolved by querying the provider with the idempotency key, not by guessing.

A strong answer mentions: the Little's law check of workers needed · a bulkhead · idempotency keys because a timeout is an unknown outcome · a single bounded retry · a breaker with fast failure · an honest fallback (pending order or second provider), not a fake success.

Your turn: spot the bugs

A team adds this to every service, in a call chain four services deep. During the next database slowdown, the database's traffic goes up eight-fold and stays high after the slowdown ends. Find the problems.

def call(url, body):
    for attempt in range(5):
        try:
            return requests.post(url, json=body)
        except Exception:
            time.sleep(0.1)
    raise ServiceUnavailable()
Compare with our answer
  • No timeout: requests waits indefinitely by default, so workers hang. Pass timeout=(0.5, 2) (connect, read).
  • Five attempts at every layer of four multiplies load by up to 5⁴ = 625 at the bottom. Retry at one layer only, with two or three attempts.
  • Fixed 100 ms sleep, no jitter: retries arrive in synchronised waves. Use exponential backoff with full jitter.
  • Retrying a POST without an idempotency key can duplicate writes.
  • Catching every exception: it retries programming errors too, but never looks at HTTP status at all, so 4xx and 5xx responses are returned as "success" while connection errors are retried. Retry only timeouts, resets and 503s; raise on others.
  • No retry budget or breaker, so during a long outage every request does all five attempts; that is why load stayed high after the slowdown ended (a metastable failure).

A strong answer mentions: the missing timeout · multiplicative retries across layers · jittered exponential backoff · idempotency for POST · retrying only transient errors · a retry budget or breaker to stop the storm.

Resilience on one page
  • Slow is worse than dead: Little's law, in-flight = rate × latency. 100/s × 30 s = 3,000 workers.
  • Timeouts on every call: connect short, request a bit above p99.9; propagate deadlines.
  • Retries: idempotent calls only (or idempotency keys); transient errors only; 2–3 attempts; exponential backoff + full jitter (sleep = random(0, min(cap, base·2ⁿ))).
  • Retries multiply across layers (3 × 3 × 3 = 27): retry at one layer, with a retry budget (~10%).
  • Circuit breaker: closed → open at ~50% failures over ≥ 20 calls → half-open after ~30 s → closed if probes succeed. Slow calls count as failures; 4xx don't.
  • Bulkhead: a capped pool per dependency. Load shedding: reject early (503) past capacity, drop expired requests, by priority.
  • Fallbacks for optional features (cache, generic, omit); fail fast and honestly for critical ones. Test fallbacks.
  • Watch for metastable failures: overload that outlives its trigger. Cut load to recover.
Interactive labPractise it: keep a page fast while its dependency is slow, failing, then recoveringOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. A page server has 40 workers and gets 80 requests per second. Each request calls a dependency that suddenly takes 5 seconds instead of 50 ms. Without a timeout, roughly how many workers would it need to keep up?

2. A client, an API gateway and a backend service each retry failed calls 3 times. The database behind the backend becomes overloaded. What is the main risk?

3. Why add random jitter to exponential backoff?

4. A circuit breaker on a recommendations client is open. What happens to calls, and why is that useful?

5. A payment call times out after 3 seconds. What is the safest next step?

6. A service has three dependencies. When one of them hangs, the whole service stops responding even though every call has a 2-second timeout. Which addition helps most?

7. During a 10-second database blip, a service's queue fills and clients retry. After the database recovers, the service stays overloaded for an hour. What best explains this?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.