Interview patterns · Design 4

Real-time updates

How servers push changes to users as they happen: polling, long polling, SSE and WebSockets compared, connection servers, pub/sub fan-out to sockets, and presence.

IntermediateInterview priority: High30 min
On this page
  1. The problem: "anything new?" ten million times
  2. Signals that you are facing it
  3. The transports
  4. How it works at scale: connection servers
  5. Fan-out to many connections
  6. Presence: who is online
  7. Worked example: sizing a chat app's real-time tier
  8. When not to use it
  9. Which case studies use it
  10. In the interview
  11. Quiz

The problem: "anything new?" ten million times

A chat app has 10 million users online at peak. The first version has each app ask the server "any new messages?" every 5 seconds. That is 2 million requests a second, and almost every answer is "no". Each request still pays for TLS, authentication, routing and a database lookup. Meanwhile a message sent just after a check waits up to 5 seconds, which feels broken in a chat.

Shorter intervals make it faster and more expensive; longer ones make it cheaper and slower. The way out is to stop asking: keep a connection open and let the server push each update when it happens. That brings its own problems: millions of open connections, finding the one connection a message should go to, and knowing who is online.

Signals that you are facing it

  • "Live" anywhere in the requirements: chat, live comments, sports scores, a driver's position on a map, stock prices, collaborative editing, multiplayer games.
  • Latency measured in hundreds of milliseconds from one user's action to another user's screen.
  • Notifications while the app is open: "show the new order on the restaurant's tablet immediately".
  • Presence: online indicators, "typing...", "seen".

Say early: "Updates need to reach users within a second, so I'll hold a persistent connection per client and push, rather than poll."

The transports

Polling, long polling and a persistent connection

TransportHowGood atWeak atUse when
Short pollingClient sends a request every N secondsSimple; works through any proxy; stateless serversWasted requests; up to N seconds of delayUpdates are rare and delay is fine (a report finishing)
Long pollingServer holds the request until there is data or a timeout (~30 s); client reconnects at onceNear-instant; plain HTTPA new request per message; servers hold many waiting requestsFallback when other options are blocked
SSE (server-sent events)One HTTP response that stays open; server writes data: lines as events occurBuilt-in browser reconnect with Last-Event-ID; plain HTTP; easy to proxyServer to client only; text onlyFeeds, scores, notifications, streaming AI output
WebSocketAn HTTP request "upgrades" to a two-way connection carrying small framesBoth directions; low overhead per message; binaryYou build reconnect, heartbeats and resume yourself; some proxies need configuringChat, games, collaboration, anything two-way

Two details trip people up. Browsers allow only about 6 HTTP/1.1 connections per domain, so several SSE streams in different tabs can starve each other; HTTP/2 multiplexes many streams over one connection and removes the problem. And load balancers close idle connections (60 seconds is a common default), so persistent connections need heartbeats, a small ping every 20–30 seconds, to stay open and to detect dead peers.

When the app is in the background or closed, none of these work on phones: the operating system suspends the connection. Deliver through push notifications (APNs for iOS, FCM for Android) instead, and let the app catch up when it reopens.

How it works at scale: connection servers

Holding a connection is cheap in CPU but costs memory, and millions of them do not fit on the app servers that run business logic. Large systems separate the two:

  • Connection servers (gateways) only hold sockets: they authenticate the client, keep the connection alive and forward messages in both directions. A well-tuned gateway holds tens of thousands to a few hundred thousand idle connections; 50,000 per server is a safe planning number.
  • Stateless services behind them do the work: store a message, update a ride, compute a score.

The hard part is the return path. When Bob should receive something, the service must reach the gateway that happens to hold Bob's connection.

Routing a message to whichever gateway holds the user

Two common ways to do it:

  • A connection registry: on connect, the gateway writes user:bob → gateway-3 in Redis with a TTL it keeps refreshing. A sender looks it up and calls gateway 3 directly. Simple and precise; the registry must be updated on every connect and disconnect.
  • Pub/sub: each gateway subscribes to a channel per connected user (or per room), and services publish to the channel. No lookups, and it extends naturally to rooms. See pub/sub.

Either way, treat the push as a doorbell, not the mailbox. Redis pub/sub, for example, delivers only to whoever is subscribed at that instant and keeps nothing. Store every message durably first, with a per-conversation sequence number; push is just the fast path. A client that reconnects asks for everything after the last sequence number it saw.

Fan-out to many connections

A message to one person is one push. A live match with a million viewers, or a group chat with 1,000 members, is a fan-out. Do it in two levels:

  1. Publish once per gateway that has interested connections, not once per viewer. With 1 M viewers spread over 20 gateways, that is 20 messages.
  2. Each gateway loops over its own local subscribers for that topic and writes to their sockets: 50,000 small writes, all in memory.

For very high rates (a score changing many times a second), coalesce: send the latest state every 100–500 ms rather than every intermediate change. See fan-out for the write-time versus read-time trade-off.

Presence: who is online

Presence looks simple and is surprisingly expensive, because every change may need to reach every friend.

  • Heartbeats with a TTL: a user is online while their entry (presence:bob, TTL 60 s) keeps being refreshed. A gateway refreshes all its users in one batched write every 30 seconds rather than one write per user.
  • Debounce: phones drop and reconnect constantly in lifts and tunnels. Wait 10–30 seconds before announcing "offline", or every blip becomes a notification to every friend.
  • Fetch, don't broadcast: rather than pushing every status change to every friend, send presence only for the people currently on screen (the open chat, the visible contact list), and subscribe to those few.

Worked example: sizing a chat app's real-time tier

  • Connections: 10 M concurrent ÷ 50,000 per gateway = 200 gateways, plus 25% headroom for failures and deploys: 250.
  • Memory: at roughly 20–50 KB per connection (buffers, TLS state, session), 50,000 connections use 1–2.5 GB per gateway: fine.
  • Heartbeats: 10 M clients pinging every 30 s = 333,000 tiny frames a second across the fleet, about 1,300 per gateway. These stay at the gateway; only the batched presence refresh (250 writes every 30 s) reaches Redis.
  • Messages: 10 M users sending 20 messages an hour each ≈ 55,000 messages a second. Each is stored, then published once to the recipient's gateway.
  • Compare polling: 2,000,000 requests a second, nearly all empty.

When not to use it

  • Updates are rare or delay is fine. An export that finishes in 10 minutes needs a poll every 30 seconds or an email, not a socket.
  • Only the server talks. If clients never send on the channel, SSE is simpler than WebSockets and goes through standard HTTP infrastructure.
  • The app is in the background. Use push notifications; a socket on a suspended phone is not connected.
  • Small scale. A single server with a few thousand WebSockets needs no gateways, registry or pub/sub.

Which case studies use it

In the interview

Your turn: live scores for a final

A sports app shows live match data: score, minute and key events. For the final, 3 million users have the match screen open. Events happen a few times a minute; the match clock updates every second. Choose the transport and design the delivery path.

Compare with our answer
  • Updates only flow server to client, so SSE (or WebSockets if the app already has them). Every viewer gets the same data, so there are no per-user routes.
  • Gateways: 3 M ÷ 50 K = 60 gateways. Each subscribes once to the match topic.
  • One producer publishes each update once; pub/sub delivers it to 60 gateways; each loops over its 50 K local connections.
  • Don't push the clock every second: send events and let the client run the clock locally, correcting it with occasional updates.
  • On reconnect, the client receives the current full state (score, minute), not a replay of everything.
  • Alternative for huge audiences: a CDN with a 1–2 s TTL and client polling, which is simpler and very cacheable.

A strong answer mentions: one-way so SSE is enough · gateway count from the connection estimate · publish once per gateway, fan out locally · send state snapshots on reconnect · avoid per-second pushes · polling through a CDN as a valid alternative.

Your turn: the requirement changes

A team's chat app uses Redis pub/sub as its only delivery path: a sent message is published to user:<id> and the recipient's gateway pushes it. Users report occasional missing messages, especially on trains. What is wrong, and what would you change?

Compare with our answer

Redis pub/sub is fire-and-forget: if the recipient's connection was down at the moment of publishing (a tunnel, a gateway restart), nobody was subscribed and the message vanished.

A strong answer mentions:

  • Store first: persist each message with a per-conversation sequence number before publishing.
  • Pub/sub remains the fast path (a doorbell), not the source of truth.
  • On reconnect, the client sends the last sequence number it has, and the server returns everything after it.
  • Delivery receipts (acks) from the client, so the server knows what to resend.
  • Push notifications for users whose app is closed.
Real-time updates on one page
  • Polling: simple but wasteful; 10 M users every 5 s = 2 M requests/s.
  • Long polling: near-instant, one request per message; a fallback.
  • SSE: one-way over HTTP, auto-reconnect with Last-Event-ID; mind the 6-connection limit without HTTP/2.
  • WebSocket: two-way, low overhead; you build heartbeats, reconnect and resume.
  • Backgrounded phones: APNs / FCM push notifications.
  • Connection servers hold sockets (~50 K each to plan with); logic stays stateless behind them.
  • Route by registry (user → gateway, TTL) or per-user pub/sub channels.
  • Store first with sequence numbers; push is the doorbell; reconnect asks "since seq N".
  • Fan-out: publish once per gateway, loop locally; coalesce fast-changing state.
  • Presence: TTL + batched heartbeats, debounce offline, only for users on screen.
Check what stuck

1. An app polls every 3 seconds for new messages, and 6 million users are online. Roughly how many requests a second does polling alone create?

2. A news site wants to push breaking headlines to readers' browsers. Readers never send anything back on the channel. Which transport fits best?

3. A chat service publishes each message to Redis pub/sub and does nothing else. Messages sometimes disappear for users on unstable networks. Why?

4. Persistent connections to an app keep dropping after about a minute of inactivity, even though both ends are healthy. What is the likely cause and fix?

5. A gateway holding 50,000 connections restarts, and the next two gateways then fall over too. What should the design add first?

6. A live stream has 1 million viewers spread over 20 gateways. How should a new chat message reach them?

7. Users complain their contacts flicker between online and offline when they ride the underground. What is the best change?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.