Handling more users · Chapter 4

Latency vs throughput

Latency versus throughput, why p99 matters more than the average, the latency numbers to know by heart, queueing near full load, and Little's law.

IntermediateInterview priority: High30 min
On this page
  1. The problem: "the dashboard says 80 ms, but users say it's slow"
  2. The idea: two different ways to be fast
  3. Measuring latency: percentiles, not averages
  4. The numbers every engineer should know
  5. Why systems slow down under load: queues
  6. Little's law: connecting the two
  7. Fighting tail latency
  8. Options and trade-offs: latency or throughput?
  9. When not to chase latency
  10. In the interview
  11. Quiz

The problem: "the dashboard says 80 ms, but users say it's slow"

A team runs a checkout API. The monitoring dashboard shows an average response time of 80 ms, comfortably inside their target. Yet support tickets keep arriving: "checkout hangs", "the spinner never stops". Last month they doubled the number of servers to fix it, and nothing changed.

Two mistakes are hiding here. The team measured the average, which hides the slow requests users complain about. And they added capacity, which raises how many requests they can handle but does not make any single request faster. To fix either, you need two words used precisely: latency and throughput.

The idea: two different ways to be fast

Latency is the length of the trip, throughput is the number of lanes

  1. There are two different ways to be fast. Latency is how long one request takes. Throughput is how many requests you can finish every second.
  2. Picture a highway. The speed limit and the distance decide how long one trip takes: that is latency. The number of lanes decides how many cars get through each hour: that is throughput.
  3. Add lanes and far more cars get through, but each trip still takes just as long. More lanes raise throughput. Only a shorter or faster road lowers latency.
  • Latency is how long one request takes, from the moment it is sent to the moment the answer arrives. Measured in milliseconds.
  • Throughput is how many requests the system completes per unit of time. Measured in requests per second (or MB/s for data).

They are related but not the same. A system can have high throughput and poor latency (a cargo ship carries a lot, slowly) or low latency and low throughput (a motorbike courier). Adding parallel capacity (servers, threads, partitions) raises throughput. Lowering latency means cutting distance (CDNs, same-region calls), work (caches, indexes, fewer round trips) or waiting (spare capacity, shorter queues).

Measuring latency: percentiles, not averages

Averages hide the slow requests

  1. Averages hide pain. If 99 requests take 50 milliseconds and one takes 3 seconds, the average still looks fine. So teams track percentiles: p50 is the typical request, and p99 is the slowest one in a hundred.

A percentile says what fraction of requests were faster than a value. p50 (the median) is the typical request. p99 means 99% of requests were faster and 1% slower. p99.9 is the slowest one in a thousand.

Worked example: why the average lies

Take 1,000 requests: 990 take 50 ms and 10 take 3,000 ms.

MeasureValueWhat it suggests
Average(990 × 50 + 10 × 3,000) / 1,000 = 79.5 ms"All good"
p5050 msTypical users are fine
p993,000 ms1 in 100 requests hangs for 3 s

The average looks healthy, but 1% of requests are terrible. And 1% is not "rare": a user who loads 50 pages in a session has a 1 − 0.99^50 ≈ 40% chance of hitting at least one slow page. Your heaviest users (the ones with the biggest carts and the most data) are usually the ones in the tail.

Tail latency gets worse with fan-out

A single page often calls many backends in parallel and must wait for the slowest. If each backend is slow 1% of the time, a request that waits for 100 of them is slow with probability 1 − 0.99^100 ≈ 63%. This is tail latency amplification: the rare slow case of each part becomes the common case of the whole. Google's paper "The Tail at Scale" made this famous.

The numbers every engineer should know

Design decisions start from rough costs. Memorise the order of magnitude, not the exact digit.

OperationRoughlyCompared with RAM
L1 CPU cache read1 ns100× faster
Main memory (RAM) read100 ns1×
In-process hash map lookupunder 1 µs~10× slower
SSD random read0.1 ms (100 µs)1,000×
Round trip inside one datacenter0.5 ms5,000×
Redis / Memcached read over the network0.2–0.5 ms2,000–5,000×
Read 1 MB sequentially from an SSD~0.5–1 ms
Indexed primary-key database read1–5 ms (10–50 ms under load)
Hard disk seek10 ms100,000×
Round trip across a continent or ocean100–150 ms1,000,000×

Three lessons fall out of this table:

  1. Memory beats disk, and local beats network. That is the whole case for caching.
  2. Round trips dominate. A page that makes 20 sequential calls inside the datacenter pays about 10 ms in network alone; batching them into one call saves most of it.
  3. Distance cannot be optimised away. Light in fibre covers about 200 km per millisecond, so London to New York (about 5,600 km) costs at least ~56 ms per round trip, in practice 70–80 ms. A new HTTPS connection needs one round trip for TCP and one or two more for TLS before the first byte of data. Serve users from nearby (CDNs, regional deployments) and reuse connections.

Why systems slow down under load: queues

Waiting time explodes as a server gets busy

  1. Why do systems slow down under load? Queues. At 50 percent busy, requests rarely wait. At 90 percent, most wait behind others. Near 100 percent, waiting time explodes.
  2. So good teams keep headroom: they run machines well below full, and add capacity before they reach about 70 to 80 percent busy.

A server busy a fraction ρ (rho, the utilisation) of the time makes new requests wait behind others. In the simplest queueing model (random arrivals, one server), the average time in the system is:

time in system ≈ service time / (1 − ρ)

UtilisationTime in system (service time 10 ms)
50%20 ms
80%50 ms
90%100 ms
95%200 ms
99%1,000 ms

Real systems differ in the details, but the curve has the same shape: flat, then a cliff. Going from 50% to 90% busy quintuples latency even though the machine still "has 10% spare". The tail suffers first, because the unlucky requests arrive just behind a burst. This is why teams keep headroom, scaling out at around 60–75% utilisation, not at 95%.

Little's law: connecting the two

Little's law links concurrency, throughput and latency for any stable system:

L = λ × W (requests in flight) = (arrival rate) × (time each one spends inside)

It needs no assumptions about the traffic pattern, which makes it the most useful formula in capacity planning.

Worked example: sizing a thread pool

An API receives 2,000 requests/s and each takes 50 ms (0.05 s) end to end.

  • In flight at any moment: L = 2,000 × 0.05 = 100 requests. With one thread per request you need at least 100 threads, plus headroom: say 150–200.
  • The downstream database now slows down and requests take 500 ms. Same traffic: L = 2,000 × 0.5 = 1,000 requests in flight. A pool of 200 threads fills up, new requests queue, and the queue makes latency worse still.
  • Turned around: 200 threads at 50 ms each can sustain at most 200 / 0.05 = 4,000 requests/s. Throughput ceiling = concurrency ÷ latency.

The second bullet is how one slow dependency takes down a healthy service: latency upstream turns into concurrency, and concurrency runs out. Timeouts and circuit breakers exist to stop that.

Fighting tail latency

TechniqueHow it worksCatch
Hedged requestsIf a reply hasn't arrived by the p95 time, send the same request to a second replica and take whichever answers firstExtra load (a few percent); only safe for reads or idempotent calls
Timeouts and fallbacksGive up on a slow backend and show the page without that widgetPartial results; must design the degraded view
Fewer fan-out callsBatch, cache, or precompute so a page touches fewer servicesMore engineering, possibly stale data
Remove pausesTune garbage collection, avoid huge objects, keep disks and CPUs below saturationOngoing operational work
HeadroomRun at 50–70% utilisationPays for idle capacity
Measuring latency honestly

Many load-testing tools send a request, wait for the reply, then send the next. When the server stalls for 2 s, the tool simply sends fewer requests during the stall, so only one slow sample is recorded instead of the hundreds that real users would have suffered. This is coordinated omission, and it makes p99 look far better than reality. Use tools that send at a fixed rate regardless of replies (wrk2, k6 with constant-arrival-rate) and record latency in histograms (HdrHistogram) so percentiles can be merged correctly.

Options and trade-offs: latency or throughput?

Many techniques trade one for the other. Know which way each one pushes.

TechniqueThroughputLatencyUse when
Batching (DB writes, Kafka producer linger.ms)Up, a lotUp, slightly (wait to fill a batch)Background pipelines, logging, analytics
More servers / partitionsUpUnchanged (lower if queues were the problem)Load is near capacity
CachingUp (less work)DownRepeated reads
CDN / regional replicasUpDown (less distance)Users are far away
CompressionUp on the networkDown on slow links, up on CPULarge text payloads
Async processing via a queueUpUser sees a fast reply; the work finishes laterThe user does not need the result now

When not to chase latency

  • Batch jobs and analytics care about throughput and total cost, not p99. A nightly report that takes 40 minutes instead of 30 rarely matters.
  • Before you have measured. Find where the time actually goes (tracing, profiling) before optimising; it is usually one slow query or one chatty loop.
  • Beyond what users perceive. Under about 100 ms a response feels instant. Shaving 5 ms off an API behind a 2-second page load buys nothing.

In the interview

Your turn: a fan-out search page

A search page queries 50 index shards in parallel and waits for all of them. Each shard has p50 = 10 ms and p99 = 200 ms. Product wants page p99 under 100 ms. What is the page's latency today, and what would you change?

Compare with our answer

With 50 shards, the chance that at least one is slower than its p99 is 1 − 0.99^50 ≈ 40%, so the page's p60 or so is already around 200 ms and its p99 is worse. Options: hedge requests to a replica of any shard that hasn't answered by ~30 ms; set a deadline (say 80 ms) and return results from the shards that answered, marking the result partial; reduce fan-out with fewer, larger shards or a two-tier index; find why shards have a 200 ms tail (GC pauses, compaction, cold caches) and fix it.

A strong answer mentions: the 1 − 0.99^N calculation · that the page waits for the slowest shard · hedged requests to replicas · a deadline with partial results · reducing fan-out · investigating the shard's own tail.

Your turn: the requirement changes

Your service handles 1,000 requests/s at 100 ms each with a pool of 150 worker threads. Marketing expects 3× traffic next month, and a new fraud check will add 200 ms to every request. How many threads (or how much concurrency) do you need, and what else would you check?

Compare with our answer

Today: L = 1,000 × 0.1 = 100 in flight, so 150 threads gives 50% headroom. Next month: 3,000 requests/s × 0.3 s = 900 in flight, so you need roughly 1,200–1,400 threads of concurrency across the fleet for similar headroom, a 9× increase from a 3× traffic rise. Better: make the fraud check asynchronous or parallel with other work so it doesn't add to latency, use non-blocking I/O so concurrency is cheap, and check that downstream connection pools (database, fraud service) can handle 900 concurrent calls.

A strong answer mentions: Little's law for both cases · that latency growth multiplies concurrency · headroom above the raw number · downstream pool limits · running the fraud check in parallel or asynchronously · timeouts on the new dependency.

Latency vs throughput on one page
  • Latency = time per request; throughput = requests per second. More servers raise throughput, not single-request speed.
  • Measure p50, p99, p99.9, never just the average. Don't average percentiles across servers.
  • Fan-out amplifies the tail: 1 − 0.99^N of requests hit at least one slow backend.
  • Numbers: RAM 100 ns · SSD 0.1 ms · DC round trip 0.5 ms · Redis 0.2–0.5 ms · DB read 1–5 ms · HDD seek 10 ms · cross-continent 100–150 ms.
  • Queueing: time ≈ service ÷ (1 − utilisation). 90% busy ≈ 10× the service time. Keep headroom (scale at 60–75%).
  • Little's law: in flight = rate × latency. Throughput ceiling = concurrency ÷ latency.
  • Slow dependency → more requests in flight → pools exhausted. Timeouts, budgets, one retry with backoff, load shedding.
  • Tail fixes: hedged requests, deadlines with partial results, fewer fan-out calls, remove GC and compaction pauses.
Interactive labPractise it: the latency playgroundOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. A team's API reports an average latency of 80 ms, but users complain about hangs. Which measurement would best reveal the problem?

2. A page calls 100 backend services in parallel and waits for all of them. Each is slower than its own p99 1% of the time. Roughly what share of page loads hit at least one slow backend?

3. A service receives 4,000 requests per second and each request takes 250 ms. Using Little's law, about how many requests are in flight at any moment?

4. A server runs at 60% utilisation with a 10 ms service time. Traffic grows until it runs at 95%. Under a simple queueing model, what happens to average response time?

5. Users in Australia see 400 ms page loads from servers in Virginia, while US users see 80 ms. The servers are 30% busy. What helps most?

6. A worker pool of 50 threads processes jobs that each take 100 ms. What is the most throughput it can sustain?

7. A downstream service slows from 50 ms to 2 s. Upstream clients retry three times on timeout. What is the most likely outcome?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.