Caching
What a cache is, how hits, misses and eviction work, the read and write strategies you will be asked about, and the failure modes (stale data, stampedes, hot keys) that make caching hard.
On this page
The problem: the same question, asked a million times
A product page, a celebrity's profile, today's top post. A small number of things are asked for again and again. If every request goes all the way to the database, three things happen:
- It is slow. An indexed lookup on a healthy database takes about 1–5 ms over the network; under heavy load, or when it has to read from disk, 10–50 ms is common.
- It is wasteful. The database does the same work and returns the same answer every time.
- It does not scale. The database becomes the busiest machine in the system, and the first one to fall over.
Why the database gets hot, and what a cache adds
- Some questions get asked over and over: the same popular post, the same product page. Asking the database every single time is slow and wasteful.
- A cache is a small, very fast memory that keeps recent answers close to the server. Checking it takes well under a millisecond. The database takes several milliseconds, and far more when it is busy.
A cache is a small, fast store that keeps copies of answers you have already computed, close to where they are needed. Reading from memory on a nearby cache server such as Redis takes about 0.2–0.5 ms, and an in-process cache takes microseconds. The trade is simple: you spend some memory and accept that copies can be out of date, and in return most reads become fast and the database gets quieter.
How a read works: hits and misses
A miss fills the cache, then the next reads are hits
- If the answer is already in the cache, that is a cache hit: fast. If not, it is a cache miss: we ask the database, then save the answer in the cache for next time.
- The share of requests answered by the cache is the hit ratio. People tend to want the same popular things, so even a small cache can answer most requests.
When the app needs a value it looks in the cache first.
- Cache hit: the value is there. Return it. Done in well under a millisecond.
- Cache miss: the value is not there. Read it from the database, store a copy in the cache, then return it. This request is a little slower than having no cache at all, because it checked the cache first.
The share of reads that are hits is the hit ratio. It decides almost everything about how useful a cache is.
Worked example: what a hit ratio buys you
Say a cache read takes 0.5 ms and a database read takes 5 ms. On a miss you pay both.
| Hit ratio | Average read time | Reads reaching the database (at 10,000 reads/s) |
|---|---|---|
| 0% (no cache) | 5 ms | 10,000 / s |
| 80% | 0.8 × 0.5 + 0.2 × 5.5 = 1.5 ms | 2,000 / s |
| 95% | 0.95 × 0.5 + 0.05 × 5.5 = 0.75 ms | 500 / s |
| 99% | 0.99 × 0.5 + 0.01 × 5.5 = 0.55 ms | 100 / s |
Two things stand out. Latency improves, but the bigger win is load: at 95% the database sees 1 in 20 reads, so it stays healthy and keeps its own latency low. And going from 95% to 99% cuts database load by another 5×, which is why people work hard on the last few percent.
High hit ratios are realistic because popularity is lopsided: a few items get most of the traffic (a "long tail" or Zipf distribution). Caching the top 1–5% of items often serves 80–95% of reads.
Where caches live
Caches at every level, from the browser to the server
- Caches live at many levels: in your browser, at a CDN near you, and in memory next to the servers, using tools like Redis. The idea is always the same: keep popular answers close.
| Layer | Example | What it holds | Typical read time |
|---|---|---|---|
| Browser / app | HTTP cache, local storage | Images, scripts, API responses for one user | ~0 ms (no network) |
| CDN | Cloudflare, CloudFront | Static files and public pages, near the user | 10–30 ms |
| In-process | A hash map inside the app server | Hot config, small lookups | microseconds |
| Distributed cache | Redis, Memcached | Shared results for all app servers | 0.2–0.5 ms |
| Database | Buffer pool | Recently read pages in memory | built in |
An in-process cache is the fastest but each server has its own copy, so they drift apart and memory is duplicated; to keep them in step, servers usually listen on a pub/sub channel and drop keys when another server announces a change. A distributed cache like Redis is shared by every server, which is why it is the default answer in system design interviews.
Reading and writing: the five strategies
The real design question is who talks to the cache and when it is updated. These names come up in every interview.
| Strategy | How it works | Good at | Weak at |
|---|---|---|---|
| Cache-aside (lazy loading) | The app checks the cache; on a miss it reads the database and fills the cache. | Simple; only caches what is actually read; cache down = slower, not broken. | First read of each item is a miss; data can go stale. |
| Read-through | Like cache-aside, but the cache library loads from the database itself. | App code stays simple. | Needs a cache that supports loaders. |
| Write-through | Every write goes to the cache and the database together. | Cache stays fresh, as long as both writes succeed. | Writes are slower; if one of the two writes fails they disagree until repaired; caches data that may never be read. |
| Write-behind (write-back) | Write to the cache now, flush to the database later in batches. | Very fast writes; absorbs bursts. | Data loss if the cache dies before flushing; complex. |
| Write-around | Writes go straight to the database; the cache fills only on reads. | Avoids filling the cache with write-once data. | Recently written data is a miss on first read. |
The most common combination in practice is cache-aside for reads plus delete-on-write: when data changes, update the database and then delete the cache entry, so the next read fetches the fresh value.
def get_product(product_id):
key = f"product:{product_id}"
cached = redis.get(key)
if cached is not None: # hit
return json.loads(cached)
product = db.query("SELECT * FROM products WHERE id = %s", product_id) # miss
redis.set(key, json.dumps(product), ex=300) # keep for 5 minutes
return product
def update_price(product_id, price):
db.execute("UPDATE products SET price = %s WHERE id = %s", price, product_id)
redis.delete(f"product:{product_id}") # next read loads the new priceProduct getProduct(long id) {
String key = "product:" + id;
String cached = redis.get(key);
if (cached != null) return Json.parse(cached, Product.class); // hit
Product p = productRepo.findById(id); // miss
redis.setex(key, 300, Json.write(p)); // keep for 5 minutes
return p;
}
void updatePrice(long id, BigDecimal price) {
productRepo.updatePrice(id, price);
redis.del("product:" + id); // delete, don't update: avoids racing writers
}When the cache is full: eviction
A cache has limited memory, so when it fills up something has to go.
LRU throws out what nobody used for the longest time
- A cache is small, so when it is full something has to go. A common rule is LRU, least recently used: throw out whatever nobody has asked for in the longest time.
| Policy | Evicts | Use when |
|---|---|---|
| LRU (least recently used) | The item untouched for the longest time | The default. Recent use predicts future use. |
| LFU (least frequently used) | The item read the fewest times | Popularity is stable over time (e.g. reference data). |
| FIFO | The oldest item added | Rarely the best; simple. |
| TTL expiry | Anything past its time to live | Always combine with one of the above. |
Redis offers these as settings (allkeys-lru, allkeys-lfu, volatile-ttl and others). Its LRU is approximate: it samples a few keys and evicts the oldest of the sample, which is far cheaper than tracking exact order. In an interview, saying "LRU with a TTL on every key" is a solid default.
Why LRU sometimes fails, and what LFU fixes
A single big scan (a nightly report reading every product once) fills an LRU cache with items nobody will read again and pushes out the genuinely popular ones. LFU-style policies resist this because one read doesn't make an item look popular. Modern caches such as Caffeine (Java) use TinyLFU: a compact frequency sketch decides whether a new item is worth admitting at all.
Keeping copies fresh
Stale data, and the two ways to fix it
- Cached answers can go out of date. If a price changes in the database, the cache may still hold the old one, and users see the wrong price. That is called stale data.
- Two common fixes. Give each entry an expiry time, called a TTL or time to live, so it refreshes regularly. Or invalidate: delete the cache entry whenever the data changes.
Every cache trades freshness for speed. When the database changes, cached copies become stale. You have two tools:
- TTL (time to live): every entry expires after a fixed time. Simple and self-healing, but data can be stale for up to the TTL.
- Invalidation: delete (or update) the entry when the data changes. Fresh, but every code path that writes the data must remember to do it, and the delete can fail.
Use both: invalidate on write for freshness, and keep a TTL as a safety net so a missed delete cannot leave a wrong value forever.
How stale is acceptable depends on the data, so decide it per type:
| Data | Acceptable staleness | Approach |
|---|---|---|
| Like counts, view counts | Seconds to minutes | TTL only |
| Product descriptions | Minutes | TTL + invalidate on edit |
| Prices, stock levels | Near zero at checkout | Cache for browsing, always read the database when buying |
| Account balance, permissions | None | Don't cache, or invalidate synchronously |
What goes wrong at scale
These are the follow-up questions that separate a good answer from a great one.
Cache stampede (thundering herd)
A popular key expires. In the same millisecond, thousands of requests miss and all hit the database to rebuild the same value. The database falls over, which causes more misses.
Fixes:
- Request coalescing / locking: the first miss takes a short lock (for example
SET key:lock 1 NX EX 5in Redis) and rebuilds; everyone else waits briefly or serves the old value. - Refresh early: rebuild a hot key in the background shortly before it expires. A neat version is probabilistic early expiration: each reader near the end of the TTL refreshes with a small probability that rises as expiry approaches, so exactly one usually does it.
- Jitter the TTL: use 300 s ± a random 30 s, so many keys don't expire together (a "cache avalanche").
Hot keys
One key (a celebrity's profile during breaking news) gets so many reads that the single cache node holding it maxes out. Redis Cluster splits keys across 16,384 hash slots, but one key always lives in one slot, so adding nodes doesn't help that key. Fixes: keep a small in-process cache of the hottest keys on each app server for a few seconds, or store several copies (profile:42#1 … #8) on different nodes and read a random one. With copies, a write must delete or update all of them.
Cache penetration
Requests for things that don't exist (a bot asking for random IDs) always miss and always hit the database. Fixes: cache the "not found" result with a short TTL, or check a Bloom filter of valid IDs first.
The cache goes down
With cache-aside, the system keeps working but every read hits the database, which may not survive the full load. Run the cache as a replicated cluster, and have a plan (rate limiting, serving degraded pages) for a cold start.
Redis or Memcached?
Redis keeps data structures (strings, hashes, lists, sorted sets, sets), supports replication, persistence to disk, Lua scripts, pub/sub and clustering. It is single-threaded per shard for commands, which makes operations atomic. Memcached is a simpler, multi-threaded key-value cache: plain strings, no persistence, no replication, very efficient for large flat caches. Most designs pick Redis because sorted sets, counters and TTLs solve many problems at once (leaderboards, rate limiters, session stores). Pick Memcached when you only need a big, simple, multi-threaded string cache.
Sizing the cache
Memory is the cost. Work it out like this: 10 M products × 20% worth caching × 2 KB per entry = 4 GB of values. Redis adds roughly 50–100 bytes of overhead per key, which matters when values are small: 100 M tiny counters at 16 bytes each are 1.6 GB of data but closer to 8 GB in Redis. Leave about 25% headroom, then pick node sizes, and add a replica per shard so a node failure doesn't empty the cache.
Warming a cold cache
A freshly started or flushed cache has a 0% hit ratio, so the database suddenly takes all reads. Before sending full traffic to a new cache cluster, pre-load the hottest keys (from logs or the old cluster), or shift traffic over gradually.
When not to cache
- Writes dominate: if data changes more often than it is read, the cache is constantly invalidated and the hit ratio is low.
- Every request is different: personalised search results with unique parameters rarely repeat.
- Correctness beats speed: money movements, inventory at the moment of purchase, security checks.
- The database is already fast enough: a cache is another system to run, monitor and keep consistent. Add one when you have measured a need.
In the interview
A team caches account balances: on a deposit they update the database, then write the new balance into Redis with no TTL. Customers sometimes see an old balance for days. List what is wrong and how you would fix it.
Compare with our answer
A strong answer mentions:
- Writing values on update allows the write-write race: two deposits finishing out of order leave the older balance cached.
- With no TTL, a wrong value never heals.
- A balance is money: show it from the database, or cache it only with a short TTL, version checks or leases, and never use the cached value to approve a payment.
- Prefer delete on write plus a TTL, and consider change-data-capture invalidation so every write path is covered.
A news site serves 50,000 article reads per second. Articles are edited a few times in the first hour, then almost never. Breaking news can send 30% of all traffic to one article. Design the caching: what layers, what strategy, what TTL, and what would you worry about?
Compare with our answer
- CDN in front for article pages: most readers are anonymous, so the whole page can be cached at the edge. A short TTL (30–60 s) plus a purge call when an editor saves keeps edits visible quickly.
- Redis behind the app servers, cache-aside for article data and comments counts, delete on edit, TTL a few minutes with jitter.
- Hot key: the breaking-news article. The CDN absorbs most of it; also keep it in an in-process cache on each app server for a few seconds, and use request coalescing so only one request rebuilds it when it expires.
- Stampede: refresh popular articles early instead of letting them expire under load.
- Staleness: a minute of staleness is fine for article text; comment counts can be eventually consistent.
A strong answer mentions: CDN for anonymous pages · purge on edit · Redis cache-aside behind it · a hot-key plan · stampede protection · how stale each kind of data may be.
- Cache = fast copy close to the reader. Redis ~0.2–0.5 ms vs an indexed database read ~1–5 ms (much more under load).
- Hit ratio decides everything: 95% → database sees 1 in 20 reads.
- Default: cache-aside + delete on write + TTL with jitter, LRU eviction, Redis cluster.
- Writes: write-through (fresh, slower writes), write-behind (fast, risk of loss), write-around (skip write-once data).
- Staleness is a per-data decision. Never trust a cached balance or stock level at checkout.
- At scale: stampede → lock or refresh early; hot key → local cache or key copies; penetration → cache "not found" or Bloom filter; avalanche → TTL jitter.
- Delete-on-write still has a read-write race: TTL always, leases/versions or CDC when it matters.
- Size it: entries × value size + ~50–100 bytes per key + 25% headroom, with a replica per shard.
1. Your service reads user profiles 20,000 times per second and updates them 50 times per second. A few seconds of staleness is fine. Which setup fits best?
Reads outnumber writes 400 to 1 and staleness is acceptable: the textbook case for cache-aside. Delete-on-update keeps it fresh and the TTL is a safety net. Write-through also works but caches every write, and write-behind risks losing updates.
2. A cache answers reads in 0.5 ms and the database in 5 ms (a miss pays both). At a 90% hit ratio and 20,000 reads per second, how many reads per second reach the database?
10% of reads miss: 20,000 × 0.1 = 2,000 per second. The average read is about 0.9 × 0.5 + 0.1 × 5.5 = 1 ms, but the 10× drop in database load is the bigger win.
3. Which statement about "update the database, then delete the cache key" is true?
Delete-on-write fixes the writer-writer ordering problem. A reader that loaded the old row before the delete can still write it back afterwards, which TTLs, leases or versions handle.
4. A popular key expires and 5,000 requests miss at the same moment, overloading the database. What is this, and what is a good fix?
Many simultaneous misses for one key is a stampede (thundering herd). Coalescing the rebuild, refreshing early and jittering TTLs all help. Key copies fix hot keys, which is a different problem.
5. Which of these is the riskiest to serve from a cache?
Stale likes, descriptions or names are harmless. Stale stock at checkout oversells. Read the source of truth for decisions that move money or inventory.
6. Bots request millions of random product IDs that do not exist, and every one misses the cache and hits the database. What helps most?
Lookups for missing data never become hits by themselves. Caching the negative result, or rejecting impossible IDs before the database, stops the leak.
7. Why can caching a small fraction of items give a high hit ratio?
Real traffic follows a long-tail distribution. Caching the top few percent of items often serves the large majority of reads.
Score 60% or more and the lesson is marked complete.