Real-time updates
How servers push changes to users as they happen: polling, long polling, SSE and WebSockets compared, connection servers, pub/sub fan-out to sockets, and presence.
On this page
The problem: "anything new?" ten million times
A chat app has 10 million users online at peak. The first version has each app ask the server "any new messages?" every 5 seconds. That is 2 million requests a second, and almost every answer is "no". Each request still pays for TLS, authentication, routing and a database lookup. Meanwhile a message sent just after a check waits up to 5 seconds, which feels broken in a chat.
Shorter intervals make it faster and more expensive; longer ones make it cheaper and slower. The way out is to stop asking: keep a connection open and let the server push each update when it happens. That brings its own problems: millions of open connections, finding the one connection a message should go to, and knowing who is online.
Signals that you are facing it
- "Live" anywhere in the requirements: chat, live comments, sports scores, a driver's position on a map, stock prices, collaborative editing, multiplayer games.
- Latency measured in hundreds of milliseconds from one user's action to another user's screen.
- Notifications while the app is open: "show the new order on the restaurant's tablet immediately".
- Presence: online indicators, "typing...", "seen".
Say early: "Updates need to reach users within a second, so I'll hold a persistent connection per client and push, rather than poll."
The transports
Polling, long polling and a persistent connection
| Transport | How | Good at | Weak at | Use when |
|---|---|---|---|---|
| Short polling | Client sends a request every N seconds | Simple; works through any proxy; stateless servers | Wasted requests; up to N seconds of delay | Updates are rare and delay is fine (a report finishing) |
| Long polling | Server holds the request until there is data or a timeout (~30 s); client reconnects at once | Near-instant; plain HTTP | A new request per message; servers hold many waiting requests | Fallback when other options are blocked |
| SSE (server-sent events) | One HTTP response that stays open; server writes data: lines as events occur | Built-in browser reconnect with Last-Event-ID; plain HTTP; easy to proxy | Server to client only; text only | Feeds, scores, notifications, streaming AI output |
| WebSocket | An HTTP request "upgrades" to a two-way connection carrying small frames | Both directions; low overhead per message; binary | You build reconnect, heartbeats and resume yourself; some proxies need configuring | Chat, games, collaboration, anything two-way |
Two details trip people up. Browsers allow only about 6 HTTP/1.1 connections per domain, so several SSE streams in different tabs can starve each other; HTTP/2 multiplexes many streams over one connection and removes the problem. And load balancers close idle connections (60 seconds is a common default), so persistent connections need heartbeats, a small ping every 20–30 seconds, to stay open and to detect dead peers.
When the app is in the background or closed, none of these work on phones: the operating system suspends the connection. Deliver through push notifications (APNs for iOS, FCM for Android) instead, and let the app catch up when it reopens.
How it works at scale: connection servers
Holding a connection is cheap in CPU but costs memory, and millions of them do not fit on the app servers that run business logic. Large systems separate the two:
- Connection servers (gateways) only hold sockets: they authenticate the client, keep the connection alive and forward messages in both directions. A well-tuned gateway holds tens of thousands to a few hundred thousand idle connections; 50,000 per server is a safe planning number.
- Stateless services behind them do the work: store a message, update a ride, compute a score.
The hard part is the return path. When Bob should receive something, the service must reach the gateway that happens to hold Bob's connection.
Routing a message to whichever gateway holds the user
Two common ways to do it:
- A connection registry: on connect, the gateway writes
user:bob → gateway-3in Redis with a TTL it keeps refreshing. A sender looks it up and calls gateway 3 directly. Simple and precise; the registry must be updated on every connect and disconnect. - Pub/sub: each gateway subscribes to a channel per connected user (or per room), and services publish to the channel. No lookups, and it extends naturally to rooms. See pub/sub.
Either way, treat the push as a doorbell, not the mailbox. Redis pub/sub, for example, delivers only to whoever is subscribed at that instant and keeps nothing. Store every message durably first, with a per-conversation sequence number; push is just the fast path. A client that reconnects asks for everything after the last sequence number it saw.
Fan-out to many connections
A message to one person is one push. A live match with a million viewers, or a group chat with 1,000 members, is a fan-out. Do it in two levels:
- Publish once per gateway that has interested connections, not once per viewer. With 1 M viewers spread over 20 gateways, that is 20 messages.
- Each gateway loops over its own local subscribers for that topic and writes to their sockets: 50,000 small writes, all in memory.
For very high rates (a score changing many times a second), coalesce: send the latest state every 100–500 ms rather than every intermediate change. See fan-out for the write-time versus read-time trade-off.
Presence: who is online
Presence looks simple and is surprisingly expensive, because every change may need to reach every friend.
- Heartbeats with a TTL: a user is online while their entry (
presence:bob, TTL 60 s) keeps being refreshed. A gateway refreshes all its users in one batched write every 30 seconds rather than one write per user. - Debounce: phones drop and reconnect constantly in lifts and tunnels. Wait 10–30 seconds before announcing "offline", or every blip becomes a notification to every friend.
- Fetch, don't broadcast: rather than pushing every status change to every friend, send presence only for the people currently on screen (the open chat, the visible contact list), and subscribe to those few.
Worked example: sizing a chat app's real-time tier
- Connections: 10 M concurrent ÷ 50,000 per gateway = 200 gateways, plus 25% headroom for failures and deploys: 250.
- Memory: at roughly 20–50 KB per connection (buffers, TLS state, session), 50,000 connections use 1–2.5 GB per gateway: fine.
- Heartbeats: 10 M clients pinging every 30 s = 333,000 tiny frames a second across the fleet, about 1,300 per gateway. These stay at the gateway; only the batched presence refresh (250 writes every 30 s) reaches Redis.
- Messages: 10 M users sending 20 messages an hour each ≈ 55,000 messages a second. Each is stored, then published once to the recipient's gateway.
- Compare polling: 2,000,000 requests a second, nearly all empty.
When not to use it
- Updates are rare or delay is fine. An export that finishes in 10 minutes needs a poll every 30 seconds or an email, not a socket.
- Only the server talks. If clients never send on the channel, SSE is simpler than WebSockets and goes through standard HTTP infrastructure.
- The app is in the background. Use push notifications; a socket on a suspended phone is not connected.
- Small scale. A single server with a few thousand WebSockets needs no gateways, registry or pub/sub.
Which case studies use it
- Design WhatsApp: gateways, per-conversation sequence numbers, offline delivery via push notifications.
- Design Uber: the driver's position pushed to the rider every few seconds.
- Design Google Docs: edits and cursors streamed over WebSockets to everyone in a document.
- Design Zoom: signalling over persistent connections, media over separate paths.
- Design food delivery: order status pushed to customer, restaurant and courier.
- The chat app concept lesson and APIs, REST, gRPC and WebSockets.
In the interview
A sports app shows live match data: score, minute and key events. For the final, 3 million users have the match screen open. Events happen a few times a minute; the match clock updates every second. Choose the transport and design the delivery path.
Compare with our answer
- Updates only flow server to client, so SSE (or WebSockets if the app already has them). Every viewer gets the same data, so there are no per-user routes.
- Gateways: 3 M ÷ 50 K = 60 gateways. Each subscribes once to the match topic.
- One producer publishes each update once; pub/sub delivers it to 60 gateways; each loops over its 50 K local connections.
- Don't push the clock every second: send events and let the client run the clock locally, correcting it with occasional updates.
- On reconnect, the client receives the current full state (score, minute), not a replay of everything.
- Alternative for huge audiences: a CDN with a 1–2 s TTL and client polling, which is simpler and very cacheable.
A strong answer mentions: one-way so SSE is enough · gateway count from the connection estimate · publish once per gateway, fan out locally · send state snapshots on reconnect · avoid per-second pushes · polling through a CDN as a valid alternative.
A team's chat app uses Redis pub/sub as its only delivery path: a sent message is published to user:<id> and the recipient's gateway pushes it. Users report occasional missing messages, especially on trains. What is wrong, and what would you change?
Compare with our answer
Redis pub/sub is fire-and-forget: if the recipient's connection was down at the moment of publishing (a tunnel, a gateway restart), nobody was subscribed and the message vanished.
A strong answer mentions:
- Store first: persist each message with a per-conversation sequence number before publishing.
- Pub/sub remains the fast path (a doorbell), not the source of truth.
- On reconnect, the client sends the last sequence number it has, and the server returns everything after it.
- Delivery receipts (acks) from the client, so the server knows what to resend.
- Push notifications for users whose app is closed.
- Polling: simple but wasteful; 10 M users every 5 s = 2 M requests/s.
- Long polling: near-instant, one request per message; a fallback.
- SSE: one-way over HTTP, auto-reconnect with Last-Event-ID; mind the 6-connection limit without HTTP/2.
- WebSocket: two-way, low overhead; you build heartbeats, reconnect and resume.
- Backgrounded phones: APNs / FCM push notifications.
- Connection servers hold sockets (~50 K each to plan with); logic stays stateless behind them.
- Route by registry (user → gateway, TTL) or per-user pub/sub channels.
- Store first with sequence numbers; push is the doorbell; reconnect asks "since seq N".
- Fan-out: publish once per gateway, loop locally; coalesce fast-changing state.
- Presence: TTL + batched heartbeats, debounce offline, only for users on screen.
1. An app polls every 3 seconds for new messages, and 6 million users are online. Roughly how many requests a second does polling alone create?
6,000,000 ÷ 3 = 2,000,000 requests a second, nearly all returning nothing. That load disappears with a persistent connection.
2. A news site wants to push breaking headlines to readers' browsers. Readers never send anything back on the channel. Which transport fits best?
SSE is built for server-to-client streams, reconnects automatically and passes through normal HTTP infrastructure. WebSockets work too but add two-way machinery that isn't needed.
3. A chat service publishes each message to Redis pub/sub and does nothing else. Messages sometimes disappear for users on unstable networks. Why?
Pub/sub only delivers to current subscribers. Messages must be stored first, with sequence numbers, so reconnecting clients can fetch what they missed.
4. Persistent connections to an app keep dropping after about a minute of inactivity, even though both ends are healthy. What is the likely cause and fix?
Load balancers and proxies close connections with no traffic (60 s is a common default). Regular pings keep them open and detect dead peers.
5. A gateway holding 50,000 connections restarts, and the next two gateways then fall over too. What should the design add first?
Every client reconnecting at once creates a storm. Jitter spreads the reconnects over time, and draining avoids causing the storm on deploys. Faster detection only makes the storm arrive sooner.
6. A live stream has 1 million viewers spread over 20 gateways. How should a new chat message reach them?
Two-level fan-out: about 20 publishes, then each gateway writes to its 50,000 local sockets in memory. Per-viewer routing or inboxes would multiply the work a million times.
7. Users complain their contacts flicker between online and offline when they ride the underground. What is the best change?
Short network drops are normal on phones. Debouncing hides them; without a TTL, crashed clients would appear online forever.
Score 60% or more and the lesson is marked complete.