Latency vs throughput
Latency versus throughput, why p99 matters more than the average, the latency numbers to know by heart, queueing near full load, and Little's law.
On this page
- The problem: "the dashboard says 80 ms, but users say it's slow"
- The idea: two different ways to be fast
- Measuring latency: percentiles, not averages
- The numbers every engineer should know
- Why systems slow down under load: queues
- Little's law: connecting the two
- Fighting tail latency
- Options and trade-offs: latency or throughput?
- When not to chase latency
- In the interview
- Quiz
The problem: "the dashboard says 80 ms, but users say it's slow"
A team runs a checkout API. The monitoring dashboard shows an average response time of 80 ms, comfortably inside their target. Yet support tickets keep arriving: "checkout hangs", "the spinner never stops". Last month they doubled the number of servers to fix it, and nothing changed.
Two mistakes are hiding here. The team measured the average, which hides the slow requests users complain about. And they added capacity, which raises how many requests they can handle but does not make any single request faster. To fix either, you need two words used precisely: latency and throughput.
The idea: two different ways to be fast
Latency is the length of the trip, throughput is the number of lanes
- There are two different ways to be fast. Latency is how long one request takes. Throughput is how many requests you can finish every second.
- Picture a highway. The speed limit and the distance decide how long one trip takes: that is latency. The number of lanes decides how many cars get through each hour: that is throughput.
- Add lanes and far more cars get through, but each trip still takes just as long. More lanes raise throughput. Only a shorter or faster road lowers latency.
- Latency is how long one request takes, from the moment it is sent to the moment the answer arrives. Measured in milliseconds.
- Throughput is how many requests the system completes per unit of time. Measured in requests per second (or MB/s for data).
They are related but not the same. A system can have high throughput and poor latency (a cargo ship carries a lot, slowly) or low latency and low throughput (a motorbike courier). Adding parallel capacity (servers, threads, partitions) raises throughput. Lowering latency means cutting distance (CDNs, same-region calls), work (caches, indexes, fewer round trips) or waiting (spare capacity, shorter queues).
Measuring latency: percentiles, not averages
Averages hide the slow requests
- Averages hide pain. If 99 requests take 50 milliseconds and one takes 3 seconds, the average still looks fine. So teams track percentiles: p50 is the typical request, and p99 is the slowest one in a hundred.
A percentile says what fraction of requests were faster than a value. p50 (the median) is the typical request. p99 means 99% of requests were faster and 1% slower. p99.9 is the slowest one in a thousand.
Worked example: why the average lies
Take 1,000 requests: 990 take 50 ms and 10 take 3,000 ms.
| Measure | Value | What it suggests |
|---|---|---|
| Average | (990 × 50 + 10 × 3,000) / 1,000 = 79.5 ms | "All good" |
| p50 | 50 ms | Typical users are fine |
| p99 | 3,000 ms | 1 in 100 requests hangs for 3 s |
The average looks healthy, but 1% of requests are terrible. And 1% is not "rare": a user who loads 50 pages in a session has a 1 − 0.99^50 ≈ 40% chance of hitting at least one slow page. Your heaviest users (the ones with the biggest carts and the most data) are usually the ones in the tail.
Tail latency gets worse with fan-out
A single page often calls many backends in parallel and must wait for the slowest. If each backend is slow 1% of the time, a request that waits for 100 of them is slow with probability 1 − 0.99^100 ≈ 63%. This is tail latency amplification: the rare slow case of each part becomes the common case of the whole. Google's paper "The Tail at Scale" made this famous.
The numbers every engineer should know
Design decisions start from rough costs. Memorise the order of magnitude, not the exact digit.
| Operation | Roughly | Compared with RAM |
|---|---|---|
| L1 CPU cache read | 1 ns | 100× faster |
| Main memory (RAM) read | 100 ns | 1× |
| In-process hash map lookup | under 1 µs | ~10× slower |
| SSD random read | 0.1 ms (100 µs) | 1,000× |
| Round trip inside one datacenter | 0.5 ms | 5,000× |
| Redis / Memcached read over the network | 0.2–0.5 ms | 2,000–5,000× |
| Read 1 MB sequentially from an SSD | ~0.5–1 ms | |
| Indexed primary-key database read | 1–5 ms (10–50 ms under load) | |
| Hard disk seek | 10 ms | 100,000× |
| Round trip across a continent or ocean | 100–150 ms | 1,000,000× |
Three lessons fall out of this table:
- Memory beats disk, and local beats network. That is the whole case for caching.
- Round trips dominate. A page that makes 20 sequential calls inside the datacenter pays about 10 ms in network alone; batching them into one call saves most of it.
- Distance cannot be optimised away. Light in fibre covers about 200 km per millisecond, so London to New York (about 5,600 km) costs at least ~56 ms per round trip, in practice 70–80 ms. A new HTTPS connection needs one round trip for TCP and one or two more for TLS before the first byte of data. Serve users from nearby (CDNs, regional deployments) and reuse connections.
Why systems slow down under load: queues
Waiting time explodes as a server gets busy
- Why do systems slow down under load? Queues. At 50 percent busy, requests rarely wait. At 90 percent, most wait behind others. Near 100 percent, waiting time explodes.
- So good teams keep headroom: they run machines well below full, and add capacity before they reach about 70 to 80 percent busy.
A server busy a fraction ρ (rho, the utilisation) of the time makes new requests wait behind others. In the simplest queueing model (random arrivals, one server), the average time in the system is:
time in system ≈ service time / (1 − ρ)
| Utilisation | Time in system (service time 10 ms) |
|---|---|
| 50% | 20 ms |
| 80% | 50 ms |
| 90% | 100 ms |
| 95% | 200 ms |
| 99% | 1,000 ms |
Real systems differ in the details, but the curve has the same shape: flat, then a cliff. Going from 50% to 90% busy quintuples latency even though the machine still "has 10% spare". The tail suffers first, because the unlucky requests arrive just behind a burst. This is why teams keep headroom, scaling out at around 60–75% utilisation, not at 95%.
Little's law: connecting the two
Little's law links concurrency, throughput and latency for any stable system:
L = λ × W (requests in flight) = (arrival rate) × (time each one spends inside)
It needs no assumptions about the traffic pattern, which makes it the most useful formula in capacity planning.
Worked example: sizing a thread pool
An API receives 2,000 requests/s and each takes 50 ms (0.05 s) end to end.
- In flight at any moment: L = 2,000 × 0.05 = 100 requests. With one thread per request you need at least 100 threads, plus headroom: say 150–200.
- The downstream database now slows down and requests take 500 ms. Same traffic: L = 2,000 × 0.5 = 1,000 requests in flight. A pool of 200 threads fills up, new requests queue, and the queue makes latency worse still.
- Turned around: 200 threads at 50 ms each can sustain at most 200 / 0.05 = 4,000 requests/s. Throughput ceiling = concurrency ÷ latency.
The second bullet is how one slow dependency takes down a healthy service: latency upstream turns into concurrency, and concurrency runs out. Timeouts and circuit breakers exist to stop that.
Fighting tail latency
| Technique | How it works | Catch |
|---|---|---|
| Hedged requests | If a reply hasn't arrived by the p95 time, send the same request to a second replica and take whichever answers first | Extra load (a few percent); only safe for reads or idempotent calls |
| Timeouts and fallbacks | Give up on a slow backend and show the page without that widget | Partial results; must design the degraded view |
| Fewer fan-out calls | Batch, cache, or precompute so a page touches fewer services | More engineering, possibly stale data |
| Remove pauses | Tune garbage collection, avoid huge objects, keep disks and CPUs below saturation | Ongoing operational work |
| Headroom | Run at 50–70% utilisation | Pays for idle capacity |
Measuring latency honestly
Many load-testing tools send a request, wait for the reply, then send the next. When the server stalls for 2 s, the tool simply sends fewer requests during the stall, so only one slow sample is recorded instead of the hundreds that real users would have suffered. This is coordinated omission, and it makes p99 look far better than reality. Use tools that send at a fixed rate regardless of replies (wrk2, k6 with constant-arrival-rate) and record latency in histograms (HdrHistogram) so percentiles can be merged correctly.
Options and trade-offs: latency or throughput?
Many techniques trade one for the other. Know which way each one pushes.
| Technique | Throughput | Latency | Use when |
|---|---|---|---|
Batching (DB writes, Kafka producer linger.ms) | Up, a lot | Up, slightly (wait to fill a batch) | Background pipelines, logging, analytics |
| More servers / partitions | Up | Unchanged (lower if queues were the problem) | Load is near capacity |
| Caching | Up (less work) | Down | Repeated reads |
| CDN / regional replicas | Up | Down (less distance) | Users are far away |
| Compression | Up on the network | Down on slow links, up on CPU | Large text payloads |
| Async processing via a queue | Up | User sees a fast reply; the work finishes later | The user does not need the result now |
When not to chase latency
- Batch jobs and analytics care about throughput and total cost, not p99. A nightly report that takes 40 minutes instead of 30 rarely matters.
- Before you have measured. Find where the time actually goes (tracing, profiling) before optimising; it is usually one slow query or one chatty loop.
- Beyond what users perceive. Under about 100 ms a response feels instant. Shaving 5 ms off an API behind a 2-second page load buys nothing.
In the interview
A search page queries 50 index shards in parallel and waits for all of them. Each shard has p50 = 10 ms and p99 = 200 ms. Product wants page p99 under 100 ms. What is the page's latency today, and what would you change?
Compare with our answer
With 50 shards, the chance that at least one is slower than its p99 is 1 − 0.99^50 ≈ 40%, so the page's p60 or so is already around 200 ms and its p99 is worse. Options: hedge requests to a replica of any shard that hasn't answered by ~30 ms; set a deadline (say 80 ms) and return results from the shards that answered, marking the result partial; reduce fan-out with fewer, larger shards or a two-tier index; find why shards have a 200 ms tail (GC pauses, compaction, cold caches) and fix it.
A strong answer mentions: the 1 − 0.99^N calculation · that the page waits for the slowest shard · hedged requests to replicas · a deadline with partial results · reducing fan-out · investigating the shard's own tail.
Your service handles 1,000 requests/s at 100 ms each with a pool of 150 worker threads. Marketing expects 3× traffic next month, and a new fraud check will add 200 ms to every request. How many threads (or how much concurrency) do you need, and what else would you check?
Compare with our answer
Today: L = 1,000 × 0.1 = 100 in flight, so 150 threads gives 50% headroom. Next month: 3,000 requests/s × 0.3 s = 900 in flight, so you need roughly 1,200–1,400 threads of concurrency across the fleet for similar headroom, a 9× increase from a 3× traffic rise. Better: make the fraud check asynchronous or parallel with other work so it doesn't add to latency, use non-blocking I/O so concurrency is cheap, and check that downstream connection pools (database, fraud service) can handle 900 concurrent calls.
A strong answer mentions: Little's law for both cases · that latency growth multiplies concurrency · headroom above the raw number · downstream pool limits · running the fraud check in parallel or asynchronously · timeouts on the new dependency.
- Latency = time per request; throughput = requests per second. More servers raise throughput, not single-request speed.
- Measure p50, p99, p99.9, never just the average. Don't average percentiles across servers.
- Fan-out amplifies the tail: 1 − 0.99^N of requests hit at least one slow backend.
- Numbers: RAM 100 ns · SSD 0.1 ms · DC round trip 0.5 ms · Redis 0.2–0.5 ms · DB read 1–5 ms · HDD seek 10 ms · cross-continent 100–150 ms.
- Queueing: time ≈ service ÷ (1 − utilisation). 90% busy ≈ 10× the service time. Keep headroom (scale at 60–75%).
- Little's law: in flight = rate × latency. Throughput ceiling = concurrency ÷ latency.
- Slow dependency → more requests in flight → pools exhausted. Timeouts, budgets, one retry with backoff, load shedding.
- Tail fixes: hedged requests, deadlines with partial results, fewer fan-out calls, remove GC and compaction pauses.
1. A team's API reports an average latency of 80 ms, but users complain about hangs. Which measurement would best reveal the problem?
Hangs live in the tail. p99 and p99.9 show the slowest 1% and 0.1% of requests, which both the average and the median hide.
2. A page calls 100 backend services in parallel and waits for all of them. Each is slower than its own p99 1% of the time. Roughly what share of page loads hit at least one slow backend?
The page is fast only if all 100 are fast: 0.99^100 ≈ 0.37. So about 63% of pages see at least one slow response. This is tail latency amplification.
3. A service receives 4,000 requests per second and each request takes 250 ms. Using Little's law, about how many requests are in flight at any moment?
L = λ × W = 4,000/s × 0.25 s = 1,000. If the pool is smaller than that, requests queue and latency grows.
4. A server runs at 60% utilisation with a 10 ms service time. Traffic grows until it runs at 95%. Under a simple queueing model, what happens to average response time?
Time in system ≈ service ÷ (1 − ρ): 10 / 0.4 = 25 ms, then 10 / 0.05 = 200 ms. Waiting explodes near full utilisation, which is why teams keep headroom.
5. Users in Australia see 400 ms page loads from servers in Virginia, while US users see 80 ms. The servers are 30% busy. What helps most?
The servers are not busy, so the time is network distance: each round trip across the Pacific costs well over 100 ms, and a page needs several. Only moving the content closer removes that.
6. A worker pool of 50 threads processes jobs that each take 100 ms. What is the most throughput it can sustain?
Throughput ceiling = concurrency ÷ latency = 50 / 0.1 s = 500 per second. To go higher, add threads or make each job faster.
7. A downstream service slows from 50 ms to 2 s. Upstream clients retry three times on timeout. What is the most likely outcome?
Little's law: 40× the latency means 40× the requests in flight, and retries add more on top. Pools fill and the failure spreads upstream. Timeouts, one retry with backoff, and load shedding contain it.
Score 60% or more and the lesson is marked complete.