Hot keys and hot partitions
Why one popular key can overload a whole cluster, how to detect it, and the fixes: local caches, key splitting, request coalescing, sharded counters and the celebrity problem.
On this page
The problem: one key, one machine
A social app caches every post in a Redis cluster of 20 shards. Capacity planning says the cluster handles 2 million operations a second, and normal load is 300,000. Then a pop star posts. Within a minute 200,000 reads a second ask for the same key, post:42.
The cluster has plenty of spare capacity, but it does not matter. Hashing sends every request for one key to the same shard. That one Redis node, good for roughly 100,000 operations a second, is overloaded. Its network is too: 200,000 reads of a 5 KB post is 1 GB/s, close to the whole capacity of a 10 Gbit/s network card. Requests for every other key on that shard slow down as well, so users who have never heard of the pop star see errors.
That is a hot key: a single key receiving a disproportionate share of traffic. A hot partition is the same thing one level up: one shard, partition or node taking far more than its share, because of one hot key or many warm ones that happen to live together. Adding shards does not help, because the key still lives on exactly one of them.
Signals that you are facing it
- Uneven popularity: celebrities, viral posts, a flash-sale product, a live match, a trending hashtag.
- A global or shared key: a site-wide counter, a config entry every request reads, "the current leaderboard".
- Skewed write targets: likes on one post, bids on one auction, stock of one item, messages in one huge group chat.
- In production: one shard much busier than the rest (CPU, network, latency), throttling errors on one partition while overall usage is low.
In an interview, say: "Popularity here is skewed, so I'll plan for hot keys on both the read path and the write path."
How to detect hot keys
You cannot fix what you can't see, and hot keys appear suddenly. Useful sources:
- Per-shard metrics: CPU, network and operation rate per node. A hot partition shows up as one node far above the median.
- Client-side sampling: app servers count the keys they request (sampling 1 in 100 is plenty) and report the top ones every few seconds. A count-min sketch (a small fixed-size table of counters that estimates how often each key appears, never undercounting) or a top-K structure keeps this cheap.
- Store tooling:
redis-cli --hotkeysscans for frequently used keys when Redis uses an LFU eviction policy; DynamoDB's CloudWatch Contributor Insights lists the most-accessed partition keys. - Known in advance: the business usually knows who has 50 million followers or which product goes on sale at noon. Mark those keys hot before the traffic arrives.
Fixing hot reads
A local cache in each app server
The cheapest fix is to stop asking. Each app server keeps very hot values in its own memory (a near cache or local cache) for 1–2 seconds. With 50 app servers, Redis then sees about 50 requests a second for post:42 instead of 200,000, and the app saves a network round trip.
One key on one shard, then key splitting, then a local cache
Catch: each server's copy can be up to one TTL stale, and different servers may disagree briefly. Keep TTLs short, and only cache keys you have detected (or marked) as hot, so memory stays small.
Key splitting (replicating the key)
Store N copies of the value under names that hash to different shards: post:42#0 to post:42#9. Each reader picks one at random, so ten shards share the load: 20,000 reads a second each. Writes must update all ten copies, which is acceptable for a value that changes rarely.
In Redis Cluster, keys are assigned to slots by hashing the part of the name inside {...} if there is one. Never write the copies as {post:42}#0: that hash tag forces all copies into the same slot, on the same shard, and the split does nothing.
Read replicas of the hot shard
Redis and most databases can serve reads from a shard's replicas. Two replicas triple the read capacity for every key on that shard, with no change to key names. Catch: replicas lag slightly, and the gain is bounded by the number of replicas.
Request coalescing
The most dangerous moment for a hot key is when it expires or is evicted. Every request misses at once and goes to the database to rebuild the same value.
A cache stampede, and coalescing misses into one rebuild
Request coalescing (also called single flight) lets only one request rebuild a key: it takes a short lock (SET lock:post:42 1 NX PX 2000 in Redis, or an in-process map of in-flight requests), and the others wait briefly and read the result. Related defences: refresh hot keys before they expire (when the remaining TTL is small, one request refreshes in the background), add random jitter to TTLs so related keys don't expire together, and serve the stale value while the rebuild runs.
Fixing hot writes
Reads can be copied; writes to one logical value must be combined somewhere. Three approaches:
| Technique | How | Catch |
|---|---|---|
| Sharded counter | Split likes:42 into 32 sub-counters on different shards; each increment picks one at random; reads sum all 32 (and cache the sum for a second) | Reads cost 32 lookups; only for values that can be summed |
| Aggregate first | Send increments to a queue; a consumer sums per key for a second and writes +N once | Seconds of delay; consumer must be idempotent |
| Serialise in one place | Route all writes for the key to one partition and one single-threaded consumer that applies them in memory, writing snapshots | One machine's speed is the cap; needs failover |
One hot row, then a sharded counter that spreads the writes
For a value that must stay exact under contention, such as stock of one product, see contention: you can split the stock into buckets (10 buckets of 500 units each), but then "sold out" means every bucket is empty.
The celebrity problem
The celebrity problem is hot keys at their most visible, and it has two faces:
- Hot reads: millions of people read the celebrity's profile and posts. Local caches, key splitting and a CDN handle it.
- Hot fan-out: when the celebrity posts, a design that copies each post into every follower's timeline must make 50 million writes. The usual answer is a hybrid: followers pull celebrity posts at read time instead. See fan-out.
Fan-out on read, then the hybrid used for celebrities
Systems often treat a known list of very large accounts as a special class with its own path: different caching, pull instead of push, separate queues so their work cannot delay everyone else's.
Worked example: the pop star's post
- Peak: 200,000 reads a second for
post:42, a 5 KB value. - Without changes: one Redis shard at 200% of its ~100 K ops/s, and 1 GB/s of network from one node.
- Local cache, TTL 1 s, 50 app servers: Redis sees ~50 reads a second for this key. Memory per server: the top 1,000 hot keys × 5 KB = 5 MB.
- If the key is not detected for the first 30 seconds, key splitting across 10 shards (or 2 replicas per shard) keeps each node at 20,000–67,000 reads a second while detection catches up.
- Likes on the post: 40,000 a second. A sharded counter with 32 sub-keys puts ~1,250 increments a second on each; the displayed count is the cached sum, at most 1 second old.
When not to use these tools
- The skew isn't there. Splitting every key multiplies writes and memory for no gain. Detect first, then treat the hot ones specially.
- The value must be exact on every read. Local caches and replicas serve slightly old data. Account balances or seat availability need a different approach.
- A CDN can do it. If the hot content is public (an image, a public post page), cache it at the edge and keep it away from your servers entirely.
Which case studies use it
- Design Twitter / X timeline and Design Instagram: celebrities, hybrid fan-out, hot posts.
- Design Ticketmaster: one popular event receives most of the traffic.
- Design a game leaderboard: one global sorted set read by everyone.
- Design YouTube: a viral video's metadata and view counter.
- Design search autocomplete: trending prefixes become hot during news events.
In the interview
A shop runs a 12:00 flash sale on one phone. 500,000 people refresh the product page, about 150,000 requests a second, and each page shows the remaining stock. Stock is in Postgres; product details are in a Redis cluster. Design the read path for the page.
Compare with our answer
- Product details (name, photos, description) rarely change: serve them from the CDN or a local cache with a TTL of seconds. Redis barely sees them.
- The stock number is hot and changing. Don't read Postgres per view: keep a cached stock figure refreshed every 1–2 seconds by one process, and show "a few left" style rounding. The exact check happens only at checkout.
- Make sure the cached stock key's refresh is coalesced (one refresher), never rebuilt by every request.
- If the Redis key itself is too hot, split it across shards or use local caching in app servers.
- Consider a waiting room in front of checkout so the write path sees a controlled rate.
A strong answer mentions: separate static details from the hot stock number · CDN or local cache for details · stock refreshed by one process, not per request · exact check only at checkout · coalescing · admission control.
A team splits a hot Redis key into ten copies named {product:7}:copy0 to {product:7}:copy9 and picks one at random per read. Monitoring shows the hot shard is exactly as busy as before. Why, and what would you change?
Compare with our answer
The braces are a Redis Cluster hash tag: only product:7 is hashed, so all ten copies land in the same slot on the same shard. The load did not move.
A strong answer mentions:
- Redis Cluster hashes only the text inside
{...}when present. - Rename the copies so the hashed part differs, e.g.
product:7:copy0(no braces) or{product:7:0}. - Verify placement with
CLUSTER KEYSLOTbefore relying on a split. - Writes must update all copies, and copies may briefly disagree.
- A local cache in the app may be a simpler fix than splitting at all.
- One key always lives on one shard: adding shards never fixes a hot key.
- Symptoms: one node's CPU or network far above the rest; throttling on one partition.
- Detect: per-shard metrics, client-side sampled top-K (count-min sketch), store tools, known celebrity lists.
- Hot reads: local cache 1–2 s (200 K/s becomes ~1/s per app server), key splitting, shard replicas, CDN.
- Never put split copies under one Redis hash tag.
- Expiry: request coalescing, early refresh, TTL jitter, serve stale.
- Hot writes: sharded counters (sum on read), aggregate then write
+N, or serialise in one consumer. - Celebrity: hot reads plus hot fan-out; pull celebrity posts at read time.
1. A Redis cluster grows from 10 to 40 shards, but latency spikes during a celebrity's posts are unchanged. Why?
A key maps to exactly one slot and one shard. More shards spread different keys, not traffic to one key. Local caching or key splitting is needed.
2. 300,000 reads a second hit one key that changes about once a minute, served by 60 app servers. Which fix removes the most load for the least work?
A 1-second local cache turns 300,000 reads into about 60 a second, at the cost of up to a second of staleness. Splitting and replicas spread the load but still serve every read from Redis.
3. A hot cache key expires every 60 seconds, and each time the database receives thousands of identical queries for a moment. What is the most direct fix?
This is a stampede on one key. Coalescing (single flight) turns thousands of rebuilds into one. A longer TTL only makes stampedes rarer, and jitter helps when many keys expire together, not one.
4. A like counter receives 60,000 increments a second for one post. Which design keeps the count accurate and the write path healthy?
Spreading increments across sub-counters divides the write load; reads sum them (and can cache the sum briefly). Local copies would lose increments, and serialising each one on one row is the bottleneck.
5. Which signal most clearly points to a hot partition rather than a cluster that is simply too small?
Skew between nodes is the signature of a hot key or partition. Evenly rising load means you need more capacity overall.
6. A team keeps a copy of hot posts in each app server's memory for 30 seconds. A user deletes a post, and others keep seeing it. Which fix fits best?
A pub/sub invalidation clears the copies within milliseconds, and a short TTL bounds the damage if a message is missed. A 30 ms TTL would remove most of the benefit of the local cache.
Score 60% or more and the lesson is marked complete.