Interview patterns · Design 3

Hot keys and hot partitions

Why one popular key can overload a whole cluster, how to detect it, and the fixes: local caches, key splitting, request coalescing, sharded counters and the celebrity problem.

AdvancedInterview priority: High30 min
On this page
  1. The problem: one key, one machine
  2. Signals that you are facing it
  3. How to detect hot keys
  4. Fixing hot reads
  5. Fixing hot writes
  6. The celebrity problem
  7. Worked example: the pop star's post
  8. When not to use these tools
  9. Which case studies use it
  10. In the interview
  11. Quiz

The problem: one key, one machine

A social app caches every post in a Redis cluster of 20 shards. Capacity planning says the cluster handles 2 million operations a second, and normal load is 300,000. Then a pop star posts. Within a minute 200,000 reads a second ask for the same key, post:42.

The cluster has plenty of spare capacity, but it does not matter. Hashing sends every request for one key to the same shard. That one Redis node, good for roughly 100,000 operations a second, is overloaded. Its network is too: 200,000 reads of a 5 KB post is 1 GB/s, close to the whole capacity of a 10 Gbit/s network card. Requests for every other key on that shard slow down as well, so users who have never heard of the pop star see errors.

That is a hot key: a single key receiving a disproportionate share of traffic. A hot partition is the same thing one level up: one shard, partition or node taking far more than its share, because of one hot key or many warm ones that happen to live together. Adding shards does not help, because the key still lives on exactly one of them.

Signals that you are facing it

  • Uneven popularity: celebrities, viral posts, a flash-sale product, a live match, a trending hashtag.
  • A global or shared key: a site-wide counter, a config entry every request reads, "the current leaderboard".
  • Skewed write targets: likes on one post, bids on one auction, stock of one item, messages in one huge group chat.
  • In production: one shard much busier than the rest (CPU, network, latency), throttling errors on one partition while overall usage is low.

In an interview, say: "Popularity here is skewed, so I'll plan for hot keys on both the read path and the write path."

How to detect hot keys

You cannot fix what you can't see, and hot keys appear suddenly. Useful sources:

  • Per-shard metrics: CPU, network and operation rate per node. A hot partition shows up as one node far above the median.
  • Client-side sampling: app servers count the keys they request (sampling 1 in 100 is plenty) and report the top ones every few seconds. A count-min sketch (a small fixed-size table of counters that estimates how often each key appears, never undercounting) or a top-K structure keeps this cheap.
  • Store tooling: redis-cli --hotkeys scans for frequently used keys when Redis uses an LFU eviction policy; DynamoDB's CloudWatch Contributor Insights lists the most-accessed partition keys.
  • Known in advance: the business usually knows who has 50 million followers or which product goes on sale at noon. Mark those keys hot before the traffic arrives.

Fixing hot reads

A local cache in each app server

The cheapest fix is to stop asking. Each app server keeps very hot values in its own memory (a near cache or local cache) for 1–2 seconds. With 50 app servers, Redis then sees about 50 requests a second for post:42 instead of 200,000, and the app saves a network round trip.

One key on one shard, then key splitting, then a local cache

Catch: each server's copy can be up to one TTL stale, and different servers may disagree briefly. Keep TTLs short, and only cache keys you have detected (or marked) as hot, so memory stays small.

Key splitting (replicating the key)

Store N copies of the value under names that hash to different shards: post:42#0 to post:42#9. Each reader picks one at random, so ten shards share the load: 20,000 reads a second each. Writes must update all ten copies, which is acceptable for a value that changes rarely.

In Redis Cluster, keys are assigned to slots by hashing the part of the name inside {...} if there is one. Never write the copies as {post:42}#0: that hash tag forces all copies into the same slot, on the same shard, and the split does nothing.

Read replicas of the hot shard

Redis and most databases can serve reads from a shard's replicas. Two replicas triple the read capacity for every key on that shard, with no change to key names. Catch: replicas lag slightly, and the gain is bounded by the number of replicas.

Request coalescing

The most dangerous moment for a hot key is when it expires or is evicted. Every request misses at once and goes to the database to rebuild the same value.

A cache stampede, and coalescing misses into one rebuild

Request coalescing (also called single flight) lets only one request rebuild a key: it takes a short lock (SET lock:post:42 1 NX PX 2000 in Redis, or an in-process map of in-flight requests), and the others wait briefly and read the result. Related defences: refresh hot keys before they expire (when the remaining TTL is small, one request refreshes in the background), add random jitter to TTLs so related keys don't expire together, and serve the stale value while the rebuild runs.

Fixing hot writes

Reads can be copied; writes to one logical value must be combined somewhere. Three approaches:

TechniqueHowCatch
Sharded counterSplit likes:42 into 32 sub-counters on different shards; each increment picks one at random; reads sum all 32 (and cache the sum for a second)Reads cost 32 lookups; only for values that can be summed
Aggregate firstSend increments to a queue; a consumer sums per key for a second and writes +N onceSeconds of delay; consumer must be idempotent
Serialise in one placeRoute all writes for the key to one partition and one single-threaded consumer that applies them in memory, writing snapshotsOne machine's speed is the cap; needs failover

One hot row, then a sharded counter that spreads the writes

For a value that must stay exact under contention, such as stock of one product, see contention: you can split the stock into buckets (10 buckets of 500 units each), but then "sold out" means every bucket is empty.

The celebrity problem

The celebrity problem is hot keys at their most visible, and it has two faces:

  • Hot reads: millions of people read the celebrity's profile and posts. Local caches, key splitting and a CDN handle it.
  • Hot fan-out: when the celebrity posts, a design that copies each post into every follower's timeline must make 50 million writes. The usual answer is a hybrid: followers pull celebrity posts at read time instead. See fan-out.

Fan-out on read, then the hybrid used for celebrities

Systems often treat a known list of very large accounts as a special class with its own path: different caching, pull instead of push, separate queues so their work cannot delay everyone else's.

Worked example: the pop star's post

  • Peak: 200,000 reads a second for post:42, a 5 KB value.
  • Without changes: one Redis shard at 200% of its ~100 K ops/s, and 1 GB/s of network from one node.
  • Local cache, TTL 1 s, 50 app servers: Redis sees ~50 reads a second for this key. Memory per server: the top 1,000 hot keys × 5 KB = 5 MB.
  • If the key is not detected for the first 30 seconds, key splitting across 10 shards (or 2 replicas per shard) keeps each node at 20,000–67,000 reads a second while detection catches up.
  • Likes on the post: 40,000 a second. A sharded counter with 32 sub-keys puts ~1,250 increments a second on each; the displayed count is the cached sum, at most 1 second old.

When not to use these tools

  • The skew isn't there. Splitting every key multiplies writes and memory for no gain. Detect first, then treat the hot ones specially.
  • The value must be exact on every read. Local caches and replicas serve slightly old data. Account balances or seat availability need a different approach.
  • A CDN can do it. If the hot content is public (an image, a public post page), cache it at the edge and keep it away from your servers entirely.

Which case studies use it

In the interview

Your turn: the flash-sale product page

A shop runs a 12:00 flash sale on one phone. 500,000 people refresh the product page, about 150,000 requests a second, and each page shows the remaining stock. Stock is in Postgres; product details are in a Redis cluster. Design the read path for the page.

Compare with our answer
  • Product details (name, photos, description) rarely change: serve them from the CDN or a local cache with a TTL of seconds. Redis barely sees them.
  • The stock number is hot and changing. Don't read Postgres per view: keep a cached stock figure refreshed every 1–2 seconds by one process, and show "a few left" style rounding. The exact check happens only at checkout.
  • Make sure the cached stock key's refresh is coalesced (one refresher), never rebuilt by every request.
  • If the Redis key itself is too hot, split it across shards or use local caching in app servers.
  • Consider a waiting room in front of checkout so the write path sees a controlled rate.

A strong answer mentions: separate static details from the hot stock number · CDN or local cache for details · stock refreshed by one process, not per request · exact check only at checkout · coalescing · admission control.

Your turn: spot the bug

A team splits a hot Redis key into ten copies named {product:7}:copy0 to {product:7}:copy9 and picks one at random per read. Monitoring shows the hot shard is exactly as busy as before. Why, and what would you change?

Compare with our answer

The braces are a Redis Cluster hash tag: only product:7 is hashed, so all ten copies land in the same slot on the same shard. The load did not move.

A strong answer mentions:

  • Redis Cluster hashes only the text inside {...} when present.
  • Rename the copies so the hashed part differs, e.g. product:7:copy0 (no braces) or {product:7:0}.
  • Verify placement with CLUSTER KEYSLOT before relying on a split.
  • Writes must update all copies, and copies may briefly disagree.
  • A local cache in the app may be a simpler fix than splitting at all.
Hot keys on one page
  • One key always lives on one shard: adding shards never fixes a hot key.
  • Symptoms: one node's CPU or network far above the rest; throttling on one partition.
  • Detect: per-shard metrics, client-side sampled top-K (count-min sketch), store tools, known celebrity lists.
  • Hot reads: local cache 1–2 s (200 K/s becomes ~1/s per app server), key splitting, shard replicas, CDN.
  • Never put split copies under one Redis hash tag.
  • Expiry: request coalescing, early refresh, TTL jitter, serve stale.
  • Hot writes: sharded counters (sum on read), aggregate then write +N, or serialise in one consumer.
  • Celebrity: hot reads plus hot fan-out; pull celebrity posts at read time.
Check what stuck

1. A Redis cluster grows from 10 to 40 shards, but latency spikes during a celebrity's posts are unchanged. Why?

2. 300,000 reads a second hit one key that changes about once a minute, served by 60 app servers. Which fix removes the most load for the least work?

3. A hot cache key expires every 60 seconds, and each time the database receives thousands of identical queries for a moment. What is the most direct fix?

4. A like counter receives 60,000 increments a second for one post. Which design keeps the count accurate and the write path healthy?

5. Which signal most clearly points to a hot partition rather than a cluster that is simply too small?

6. A team keeps a copy of hot posts in each app server's memory for 30 seconds. A user deletes a post, and others keep seeing it. Which fix fits best?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.