Handling more users · Chapter 2

Caching

What a cache is, how hits, misses and eviction work, the read and write strategies you will be asked about, and the failure modes (stale data, stampedes, hot keys) that make caching hard.

IntermediateInterview priority: High40 min
On this page
  1. The problem: the same question, asked a million times
  2. How a read works: hits and misses
  3. Where caches live
  4. Reading and writing: the five strategies
  5. When the cache is full: eviction
  6. Keeping copies fresh
  7. What goes wrong at scale
  8. When not to cache
  9. In the interview
  10. Quiz

The problem: the same question, asked a million times

A product page, a celebrity's profile, today's top post. A small number of things are asked for again and again. If every request goes all the way to the database, three things happen:

  • It is slow. An indexed lookup on a healthy database takes about 1–5 ms over the network; under heavy load, or when it has to read from disk, 10–50 ms is common.
  • It is wasteful. The database does the same work and returns the same answer every time.
  • It does not scale. The database becomes the busiest machine in the system, and the first one to fall over.

Why the database gets hot, and what a cache adds

  1. Some questions get asked over and over: the same popular post, the same product page. Asking the database every single time is slow and wasteful.
  2. A cache is a small, very fast memory that keeps recent answers close to the server. Checking it takes well under a millisecond. The database takes several milliseconds, and far more when it is busy.

A cache is a small, fast store that keeps copies of answers you have already computed, close to where they are needed. Reading from memory on a nearby cache server such as Redis takes about 0.2–0.5 ms, and an in-process cache takes microseconds. The trade is simple: you spend some memory and accept that copies can be out of date, and in return most reads become fast and the database gets quieter.

How a read works: hits and misses

A miss fills the cache, then the next reads are hits

  1. If the answer is already in the cache, that is a cache hit: fast. If not, it is a cache miss: we ask the database, then save the answer in the cache for next time.
  2. The share of requests answered by the cache is the hit ratio. People tend to want the same popular things, so even a small cache can answer most requests.

When the app needs a value it looks in the cache first.

  1. Cache hit: the value is there. Return it. Done in well under a millisecond.
  2. Cache miss: the value is not there. Read it from the database, store a copy in the cache, then return it. This request is a little slower than having no cache at all, because it checked the cache first.

The share of reads that are hits is the hit ratio. It decides almost everything about how useful a cache is.

Worked example: what a hit ratio buys you

Say a cache read takes 0.5 ms and a database read takes 5 ms. On a miss you pay both.

Hit ratioAverage read timeReads reaching the database (at 10,000 reads/s)
0% (no cache)5 ms10,000 / s
80%0.8 × 0.5 + 0.2 × 5.5 = 1.5 ms2,000 / s
95%0.95 × 0.5 + 0.05 × 5.5 = 0.75 ms500 / s
99%0.99 × 0.5 + 0.01 × 5.5 = 0.55 ms100 / s

Two things stand out. Latency improves, but the bigger win is load: at 95% the database sees 1 in 20 reads, so it stays healthy and keeps its own latency low. And going from 95% to 99% cuts database load by another 5×, which is why people work hard on the last few percent.

High hit ratios are realistic because popularity is lopsided: a few items get most of the traffic (a "long tail" or Zipf distribution). Caching the top 1–5% of items often serves 80–95% of reads.

Where caches live

Caches at every level, from the browser to the server

  1. Caches live at many levels: in your browser, at a CDN near you, and in memory next to the servers, using tools like Redis. The idea is always the same: keep popular answers close.
LayerExampleWhat it holdsTypical read time
Browser / appHTTP cache, local storageImages, scripts, API responses for one user~0 ms (no network)
CDNCloudflare, CloudFrontStatic files and public pages, near the user10–30 ms
In-processA hash map inside the app serverHot config, small lookupsmicroseconds
Distributed cacheRedis, MemcachedShared results for all app servers0.2–0.5 ms
DatabaseBuffer poolRecently read pages in memorybuilt in

An in-process cache is the fastest but each server has its own copy, so they drift apart and memory is duplicated; to keep them in step, servers usually listen on a pub/sub channel and drop keys when another server announces a change. A distributed cache like Redis is shared by every server, which is why it is the default answer in system design interviews.

Reading and writing: the five strategies

The real design question is who talks to the cache and when it is updated. These names come up in every interview.

StrategyHow it worksGood atWeak at
Cache-aside (lazy loading)The app checks the cache; on a miss it reads the database and fills the cache.Simple; only caches what is actually read; cache down = slower, not broken.First read of each item is a miss; data can go stale.
Read-throughLike cache-aside, but the cache library loads from the database itself.App code stays simple.Needs a cache that supports loaders.
Write-throughEvery write goes to the cache and the database together.Cache stays fresh, as long as both writes succeed.Writes are slower; if one of the two writes fails they disagree until repaired; caches data that may never be read.
Write-behind (write-back)Write to the cache now, flush to the database later in batches.Very fast writes; absorbs bursts.Data loss if the cache dies before flushing; complex.
Write-aroundWrites go straight to the database; the cache fills only on reads.Avoids filling the cache with write-once data.Recently written data is a miss on first read.

The most common combination in practice is cache-aside for reads plus delete-on-write: when data changes, update the database and then delete the cache entry, so the next read fetches the fresh value.

def get_product(product_id):
    key = f"product:{product_id}"
    cached = redis.get(key)
    if cached is not None:              # hit
        return json.loads(cached)
    product = db.query("SELECT * FROM products WHERE id = %s", product_id)  # miss
    redis.set(key, json.dumps(product), ex=300)    # keep for 5 minutes
    return product

def update_price(product_id, price):
    db.execute("UPDATE products SET price = %s WHERE id = %s", price, product_id)
    redis.delete(f"product:{product_id}")          # next read loads the new price

When the cache is full: eviction

A cache has limited memory, so when it fills up something has to go.

LRU throws out what nobody used for the longest time

  1. A cache is small, so when it is full something has to go. A common rule is LRU, least recently used: throw out whatever nobody has asked for in the longest time.
PolicyEvictsUse when
LRU (least recently used)The item untouched for the longest timeThe default. Recent use predicts future use.
LFU (least frequently used)The item read the fewest timesPopularity is stable over time (e.g. reference data).
FIFOThe oldest item addedRarely the best; simple.
TTL expiryAnything past its time to liveAlways combine with one of the above.

Redis offers these as settings (allkeys-lru, allkeys-lfu, volatile-ttl and others). Its LRU is approximate: it samples a few keys and evicts the oldest of the sample, which is far cheaper than tracking exact order. In an interview, saying "LRU with a TTL on every key" is a solid default.

Why LRU sometimes fails, and what LFU fixes

A single big scan (a nightly report reading every product once) fills an LRU cache with items nobody will read again and pushes out the genuinely popular ones. LFU-style policies resist this because one read doesn't make an item look popular. Modern caches such as Caffeine (Java) use TinyLFU: a compact frequency sketch decides whether a new item is worth admitting at all.

Keeping copies fresh

Stale data, and the two ways to fix it

  1. Cached answers can go out of date. If a price changes in the database, the cache may still hold the old one, and users see the wrong price. That is called stale data.
  2. Two common fixes. Give each entry an expiry time, called a TTL or time to live, so it refreshes regularly. Or invalidate: delete the cache entry whenever the data changes.

Every cache trades freshness for speed. When the database changes, cached copies become stale. You have two tools:

  • TTL (time to live): every entry expires after a fixed time. Simple and self-healing, but data can be stale for up to the TTL.
  • Invalidation: delete (or update) the entry when the data changes. Fresh, but every code path that writes the data must remember to do it, and the delete can fail.

Use both: invalidate on write for freshness, and keep a TTL as a safety net so a missed delete cannot leave a wrong value forever.

How stale is acceptable depends on the data, so decide it per type:

DataAcceptable stalenessApproach
Like counts, view countsSeconds to minutesTTL only
Product descriptionsMinutesTTL + invalidate on edit
Prices, stock levelsNear zero at checkoutCache for browsing, always read the database when buying
Account balance, permissionsNoneDon't cache, or invalidate synchronously

What goes wrong at scale

These are the follow-up questions that separate a good answer from a great one.

Cache stampede (thundering herd)

A popular key expires. In the same millisecond, thousands of requests miss and all hit the database to rebuild the same value. The database falls over, which causes more misses.

Fixes:

  • Request coalescing / locking: the first miss takes a short lock (for example SET key:lock 1 NX EX 5 in Redis) and rebuilds; everyone else waits briefly or serves the old value.
  • Refresh early: rebuild a hot key in the background shortly before it expires. A neat version is probabilistic early expiration: each reader near the end of the TTL refreshes with a small probability that rises as expiry approaches, so exactly one usually does it.
  • Jitter the TTL: use 300 s ± a random 30 s, so many keys don't expire together (a "cache avalanche").

Hot keys

One key (a celebrity's profile during breaking news) gets so many reads that the single cache node holding it maxes out. Redis Cluster splits keys across 16,384 hash slots, but one key always lives in one slot, so adding nodes doesn't help that key. Fixes: keep a small in-process cache of the hottest keys on each app server for a few seconds, or store several copies (profile:42#1 … #8) on different nodes and read a random one. With copies, a write must delete or update all of them.

Cache penetration

Requests for things that don't exist (a bot asking for random IDs) always miss and always hit the database. Fixes: cache the "not found" result with a short TTL, or check a Bloom filter of valid IDs first.

The cache goes down

With cache-aside, the system keeps working but every read hits the database, which may not survive the full load. Run the cache as a replicated cluster, and have a plan (rate limiting, serving degraded pages) for a cold start.

Redis or Memcached?

Redis keeps data structures (strings, hashes, lists, sorted sets, sets), supports replication, persistence to disk, Lua scripts, pub/sub and clustering. It is single-threaded per shard for commands, which makes operations atomic. Memcached is a simpler, multi-threaded key-value cache: plain strings, no persistence, no replication, very efficient for large flat caches. Most designs pick Redis because sorted sets, counters and TTLs solve many problems at once (leaderboards, rate limiters, session stores). Pick Memcached when you only need a big, simple, multi-threaded string cache.

Sizing the cache

Memory is the cost. Work it out like this: 10 M products × 20% worth caching × 2 KB per entry = 4 GB of values. Redis adds roughly 50–100 bytes of overhead per key, which matters when values are small: 100 M tiny counters at 16 bytes each are 1.6 GB of data but closer to 8 GB in Redis. Leave about 25% headroom, then pick node sizes, and add a replica per shard so a node failure doesn't empty the cache.

Warming a cold cache

A freshly started or flushed cache has a 0% hit ratio, so the database suddenly takes all reads. Before sending full traffic to a new cache cluster, pre-load the hottest keys (from logs or the old cluster), or shift traffic over gradually.

When not to cache

  • Writes dominate: if data changes more often than it is read, the cache is constantly invalidated and the hit ratio is low.
  • Every request is different: personalised search results with unique parameters rarely repeat.
  • Correctness beats speed: money movements, inventory at the moment of purchase, security checks.
  • The database is already fast enough: a cache is another system to run, monitor and keep consistent. Add one when you have measured a need.

In the interview

Your turn: spot the bug

A team caches account balances: on a deposit they update the database, then write the new balance into Redis with no TTL. Customers sometimes see an old balance for days. List what is wrong and how you would fix it.

Compare with our answer

A strong answer mentions:

  • Writing values on update allows the write-write race: two deposits finishing out of order leave the older balance cached.
  • With no TTL, a wrong value never heals.
  • A balance is money: show it from the database, or cache it only with a short TTL, version checks or leases, and never use the cached value to approve a payment.
  • Prefer delete on write plus a TTL, and consider change-data-capture invalidation so every write path is covered.
Your turn: a news site

A news site serves 50,000 article reads per second. Articles are edited a few times in the first hour, then almost never. Breaking news can send 30% of all traffic to one article. Design the caching: what layers, what strategy, what TTL, and what would you worry about?

Compare with our answer
  • CDN in front for article pages: most readers are anonymous, so the whole page can be cached at the edge. A short TTL (30–60 s) plus a purge call when an editor saves keeps edits visible quickly.
  • Redis behind the app servers, cache-aside for article data and comments counts, delete on edit, TTL a few minutes with jitter.
  • Hot key: the breaking-news article. The CDN absorbs most of it; also keep it in an in-process cache on each app server for a few seconds, and use request coalescing so only one request rebuilds it when it expires.
  • Stampede: refresh popular articles early instead of letting them expire under load.
  • Staleness: a minute of staleness is fine for article text; comment counts can be eventually consistent.

A strong answer mentions: CDN for anonymous pages · purge on edit · Redis cache-aside behind it · a hot-key plan · stampede protection · how stale each kind of data may be.

Caching on one page
  • Cache = fast copy close to the reader. Redis ~0.2–0.5 ms vs an indexed database read ~1–5 ms (much more under load).
  • Hit ratio decides everything: 95% → database sees 1 in 20 reads.
  • Default: cache-aside + delete on write + TTL with jitter, LRU eviction, Redis cluster.
  • Writes: write-through (fresh, slower writes), write-behind (fast, risk of loss), write-around (skip write-once data).
  • Staleness is a per-data decision. Never trust a cached balance or stock level at checkout.
  • At scale: stampede → lock or refresh early; hot key → local cache or key copies; penetration → cache "not found" or Bloom filter; avalanche → TTL jitter.
  • Delete-on-write still has a read-write race: TTL always, leases/versions or CDC when it matters.
  • Size it: entries × value size + ~50–100 bytes per key + 25% headroom, with a replica per shard.
Interactive labPractise it: the caching playgroundOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. Your service reads user profiles 20,000 times per second and updates them 50 times per second. A few seconds of staleness is fine. Which setup fits best?

2. A cache answers reads in 0.5 ms and the database in 5 ms (a miss pays both). At a 90% hit ratio and 20,000 reads per second, how many reads per second reach the database?

3. Which statement about "update the database, then delete the cache key" is true?

4. A popular key expires and 5,000 requests miss at the same moment, overloading the database. What is this, and what is a good fix?

5. Which of these is the riskiest to serve from a cache?

6. Bots request millions of random product IDs that do not exist, and every one misses the cache and hits the database. What helps most?

7. Why can caching a small fraction of items give a high hit ratio?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.