Interview patterns · Design 5

Fan-out

How one event reaches many readers: fan-out on write, fan-out on read and the hybrid, with worked numbers for feeds, the celebrity problem and what each choice costs.

AdvancedInterview priority: High30 min
On this page
  1. The problem: one post, many timelines
  2. Signals that you are facing it
  3. Fan-out on write (push)
  4. Fan-out on read (pull)
  5. Hybrid: push for most, pull for the few
  6. Worked example: the numbers for the feed
  7. Fan-out beyond feeds
  8. When not to use fan-out on write
  9. Which case studies use it
  10. In the interview
  11. Quiz

The problem: one post, many timelines

A social app has 200 million daily users. Each follows about 200 accounts, and they post 100 million times a day (about 1,200 posts a second on average, 5,000 at peak). Opening the app shows a home timeline: the newest posts from everyone you follow, ready in under 200 ms.

The naive query is "fetch the latest posts from the 200 accounts I follow, merge them by time, return the top 50". At 25,000 timeline loads a second, that is 5 million post-list lookups a second, every one scattered across shards, and the slowest of the 200 lookups decides the page time.

Fan-out is the general shape of this problem: one event (a post, a message, a notification) must reach many recipients (followers, group members, subscribers). The core decision is when you do the copying: when the event is written, or when each recipient reads.

Signals that you are facing it

  • Follow graphs: feeds, timelines, "posts from people you follow".
  • Group delivery: group chats, channels, mailing lists, notifications to everyone watching an item.
  • Skewed audience sizes: most accounts have a few hundred followers; a few have tens of millions.
  • Read-heavy with a latency target on the aggregated view: "the feed must load in 200 ms".

Say early: "This is a fan-out problem. I'll decide between pushing at write time and pulling at read time based on the follower distribution, and I expect a hybrid."

Fan-out on write (push)

When someone posts, a background job copies the post ID into a stored timeline for each follower, usually a capped list in Redis (the newest few hundred IDs per user). Reading a timeline is one lookup of a precomputed list, then fetching the posts by ID, mostly from cache.

Fan-out on write, and what a celebrity does to it

  • Good at: very fast, predictable reads. Work is done once per post, not once per view.
  • Weak at: posting cost grows with follower count; a 50-million-follower account makes 50 million inserts per post. Storage for timelines nobody reads (inactive users). Deletes and unfollows must be removed from many lists.

Fan-out on read (pull)

Nothing is copied at post time. Each author has a list of their own recent posts. When a user opens the app, the timeline service fetches the recent posts of everyone they follow, merges and sorts them.

Fan-out on read: gather and merge when the user asks

  • Good at: posting is one write, whatever the audience. No storage for precomputed timelines. Unfollows and deletes take effect immediately.
  • Weak at: each read touches many lists (one per followee), so read cost and tail latency grow with the number of accounts followed. Repeated views redo the same work unless cached.

Hybrid: push for most, pull for the few

Real feeds combine them. Ordinary accounts (say, under 100,000 followers) are pushed into followers' timelines. Accounts above the threshold are not fanned out; their posts are pulled and merged in when a follower reads.

Hybrid: a precomputed timeline plus a few celebrity lists

Most users follow only a handful of very large accounts, so the read merges one precomputed list with a few celebrity lists: a handful of lookups, not hundreds. The threshold is a tuning knob, not a law; it depends on how expensive writes are compared with reads in your system.

Comparing the options

ApproachCost per postCost per timeline readGood atWeak atUse when
Fan-out on write1 insert per follower1 list lookupFast, cheap readsHuge accounts; wasted work for inactive usersReads far outnumber posts, audiences are small to medium
Fan-out on read1 insert1 lookup per followee, plus a mergeCheap writes, instant deletesSlow reads for users who follow manyAudiences are huge, or reads are rare
HybridPush only for non-celebrities1 lookup + a few celebrity listsBoth, within limitsTwo code paths; the threshold to tuneSkewed follower counts: almost every social feed

Worked example: the numbers for the feed

Assume 1,200 posts a second on average, 200 followers on average, 25,000 timeline reads a second, each user following 200 accounts.

Fan-out on writeFan-out on read
Work per second from posts1,200 × 200 = 240,000 inserts/s1,200 inserts/s
Work per second from reads25,000 lookups/s25,000 × 200 = 5,000,000 lookups/s
Read latencyOne Redis call (~1 ms)Slowest of 200 calls, plus merge

Push wins by a factor of about 20 in total operations, and gives far better read latency, because reads outnumber posts and each read would otherwise touch 200 lists.

Now the celebrity. An account with 50 million followers posts. Pushing means 50 million inserts. At a fan-out capacity of 1 million inserts a second, that one post takes 50 seconds of the whole fan-out fleet, and the last followers see it nearly a minute late. If it posts ten times during an event, ordinary users' posts queue behind it. With the hybrid, the post is one insert, and the cost moves to reads: each of the celebrity's active followers fetches one extra list when loading the timeline.

Storage: a timeline of 800 IDs × 8 bytes is 6.4 KB per user. For 200 million users that is 1.3 TB of raw data, several times that with Redis overhead and replicas. So timelines are kept only for active users (say, seen in the last 30 days); for a returning inactive user, the timeline is rebuilt on demand by pulling, then kept up to date by push again.

Fan-out beyond feeds

The same choice appears elsewhere:

  • Group chat: a message to a 1,000-member group is stored once and delivered to each member (fan-out on write to inboxes or to their connections). WhatsApp caps group size, which caps the fan-out.
  • Notifications: "someone commented on a post you follow" pushes to each watcher; very large audiences go through batch jobs and notification services with rate limits.
  • Live streams and broadcasts: fan-out happens at delivery, through connection servers, one publish per server (see real-time updates).
  • Email newsletters: always a batch job, throttled to what the sending provider allows.

When not to use fan-out on write

  • Audiences are huge for most senders: a system where every account is a broadcaster (like a news site's subscriptions) suits pull plus caching.
  • Reads are rare: if most users check once a week, precomputing timelines wastes most of the work.
  • Order or filters change at read time: a ranked feed that depends heavily on who is looking and what's trending still needs a read-time ranking step over the candidates.
  • Small scale: with a few thousand users, a single indexed query over follows and posts is fast enough.

Which case studies use it

In the interview

Your turn: a company chat with huge channels

A workplace chat tool has channels from 3 members to 200,000 members (the "all company" channel at a huge employer). Each user's sidebar shows unread counts per channel, and messages must appear within a second for people who have the channel open. Where do you push, and where do you pull?

Compare with our answer
  • Messages: store once per channel with a sequence number. Don't copy messages into per-user inboxes; reading a channel is a range query on the channel's messages (pull).
  • Live delivery: for members who are connected, publish once per connection server, which pushes to its local members (fan-out at delivery).
  • Unread counts: don't increment 200,000 counters per message. Store each user's last-read sequence number per channel; unread = latest seq − last read. That is pull, and costs nothing at write time.
  • Small channels and direct messages can push per-user notifications (mobile push) freely; huge channels batch or mute them by default.
  • Rate-limit who can post in very large channels.

A strong answer mentions: store once per channel, not per user · push only to connected members, once per gateway · unread counts from last-read sequence numbers · special handling for huge channels · mobile notification limits.

Your turn: the requirement changes

The feed was built as pure fan-out on write. Product now wants the ranked feed: show the most relevant posts first, using signals like likes in the last hour and how often you interact with the author. What changes in the fan-out design?

Compare with our answer
  • Fan-out on write still produces the candidate set: the last few hundred post IDs from accounts you follow, plus pulled celebrity posts.
  • Ranking happens at read time, because relevance changes by the minute (likes accumulate) and depends on the viewer. A ranking service scores a few hundred candidates in tens of milliseconds.
  • Signals (like counts, your affinity with each author) come from fast stores, precomputed per author or per pair, not computed per request from raw data.
  • Cache the ranked page for a short time so scrolling and refreshes don't re-rank everything.
  • Candidates can also come from outside the follow graph (recommendations), merged in the same step.

A strong answer mentions: fan-out generates candidates, ranking runs at read · relevance changes over time so it can't be precomputed at post time · precomputed features for speed · a latency budget for ranking · short-lived cache of the ranked result.

Fan-out on one page
  • Fan-out: one event to many recipients. Decide when to copy: at write (push) or at read (pull).
  • Push: inserts = posts × followers; reads = one list lookup. Best when reads dominate and audiences are modest.
  • Pull: inserts = posts; reads = one lookup per followee + merge. Best for huge audiences or rare reads.
  • Feed numbers: 1.2 K posts/s × 200 = 240 K inserts/s (push) vs 25 K reads × 200 = 5 M lookups/s (pull).
  • Celebrity: 50 M followers = 50 M inserts per post. Hybrid: pull accounts above a threshold.
  • Timelines: IDs only, capped (~800), active users only; rebuild for returning users.
  • Deletes and blocks: filter at read; clean up lazily.
  • Big jobs: chunk, separate queues, active followers first.
  • Ranking: push builds candidates, rank at read.
Check what stuck

1. A feed service has 2,000 posts a second, 300 followers per author on average, and 40,000 timeline reads a second, each user following 300 accounts. How many operations a second does fan-out on write cost for posts?

2. Why do large social networks pull a celebrity's posts at read time instead of pushing them?

3. With fan-out on write, a user deletes a post that has been copied into 2 million timelines. What is the most practical approach?

4. A newsletter platform has 5,000 writers, each with 50,000 to 2 million subscribers, and readers open the app about once a week. Which approach suits it best?

5. Ordinary users' posts take a minute to appear for followers whenever one account with 4 million followers posts. Which fix targets this directly?

6. Each user's timeline in Redis is 800 IDs of 8 bytes. Why do designs usually keep timelines only for recently active users?

7. A user follows 5,000 accounts, and with pure fan-out on read their feed takes 3 seconds. Which fix fits best?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.