Handling more users · Chapter 1

Scaling and load balancing

Vertical vs horizontal scaling, how load balancers spread traffic (algorithms, L4 vs L7, health checks), stateless servers and sessions, and sizing a fleet.

IntermediateInterview priority: High45 min
On this page
  1. The problem: one server, and the traffic keeps growing
  2. The idea: scale up, or scale out
  3. How it works: the load balancer
  4. Stateless servers: the rule that makes scaling out work
  5. Worked example: how many servers?
  6. Options and trade-offs: up or out?
  7. When not to scale out (yet)
  8. In the interview
  9. Quiz

The problem: one server, and the traffic keeps growing

Your shop's API runs on one server. Load-tested, it handles about 200 requests per second before it slows to a crawl. On a normal day you get 50. Then a marketing email goes out and 300 requests per second arrive. The server's processors are at 100%, requests queue up, time out and fail. Customers see errors at exactly the moment you paid to attract them.

One server, three times more traffic than it can take

  1. A server can only handle so many requests each second. Ours handles about 200. As more people show up, it gets busier and busier, until it simply cannot keep up.

There are only two ways to get more capacity: make the server bigger, or add more servers. Each has a different cost, and the second needs a new component, the load balancer, plus a change to how you write your application.

The idea: scale up, or scale out

Scaling up works, until it doesn't

  1. Option one: scale up. Swap it for a bigger, more powerful machine, with more processors and more memory. Now it handles 400 requests a second.
  2. Scaling up is simple, but it has limits. Bigger machines get expensive fast, there is a largest machine you can buy, and it is still just one machine. If it breaks, everything stops.

Scaling up (vertical scaling) means moving to a bigger machine: more processor cores, more memory, faster disks. Nothing in your code changes. Cloud providers rent machines with hundreds of cores and several terabytes of memory, so the ceiling is high. But:

  • There is still a ceiling, and the very largest machines cost more per unit of capacity than mid-sized ones.
  • It is still one machine. When it crashes or is patched, everything stops. It is a single point of failure: one part whose failure takes the whole system down.
  • Resizing usually means a restart, which is downtime unless you have a second machine to fail over to.
  • Software doesn't always use the extra cores. A lock that every request needs, or a single-threaded step, caps the speed-up no matter how many cores you add.

Scaling out: more ordinary servers, but who sends them traffic?

  1. Option two: scale out. Instead of one huge machine, use several ordinary ones. Need more capacity? Just add another server.
  2. But users only know one address. Without help, every request still goes to the same server, which is overloaded, while the other two sit idle.

Scaling out (horizontal scaling) means running several ordinary machines side by side and sharing the work between them. Capacity grows by adding machines, and losing one only loses a share of capacity. But users know one address, so something must stand in front and divide the requests.

How it works: the load balancer

The load balancer receives every request and passes it on

  1. Enter the load balancer. It sits in front of the servers, receives every request, and passes each one on to a server that can take it.

A load balancer (LB) is a server (or managed service) that receives every incoming request and forwards it to one of a pool of backend servers. Clients only ever see the load balancer's address, usually through a DNS name such as api.shop.com. The backends can be added, removed and replaced without clients knowing.

A load balancer typically also:

  • Terminates TLS: it does the HTTPS encryption work once, so backends receive plain HTTP inside the private network.
  • Health-checks backends and stops sending traffic to broken ones.
  • Drains connections during deploys: a server being removed gets no new requests but is allowed to finish the ones it has (often 30–300 s).

Layer 4 or layer 7?

The "layer" refers to how much of the request the balancer reads. Layer 4 (the transport layer) sees only IP addresses and ports of a TCP or UDP connection. Layer 7 (the application layer) reads the HTTP request itself: the path, headers and cookies.

Layer 4 (TCP/UDP)Layer 7 (HTTP)
Decides byIP address and port, once per connectionEach request's path, host, headers, cookies
Good atVery high throughput, low added latency (microseconds); any protocol; long-lived connections such as WebSockets or database trafficRouting /api and /images to different pools; TLS termination; retries; per-request balancing over one connection; adding headers
Weak atCan't route by URL or retry a failed HTTP request; one heavy connection stays on one serverMore CPU per request; adds roughly a millisecond; must understand the protocol
ExamplesAWS Network Load Balancer, Google Maglev, Linux IPVS, HAProxy in TCP modeNginx, Envoy, HAProxy in HTTP mode, AWS Application Load Balancer

Large sites often use both: a layer 4 tier spreads raw connections across a fleet of layer 7 proxies, which then route each HTTP request to the right service.

Choosing a server: the algorithms

Taking turns, or picking the least busy

  1. How does it choose? Round robin simply takes turns: one, two, three, one, two, three. Least connections picks whichever server is least busy right now, which works better when some requests take longer.
AlgorithmHow it picksGood atWeak atUse when
Round robinTakes turns: 1, 2, 3, 1, 2, 3Simple, even spreadIgnores server size and request costIdentical servers, similar requests
Weighted round robinTurns in proportion to a weight (a 2× machine gets 2 shares)Mixed server sizesWeights are static; slow requests still pile upKnown, stable differences in capacity
Least connectionsServer with the fewest requests in progressRequests of very different lengthsNeeds a live count per server; can be fooled (see below)Mixed fast and slow requests
Power of two choicesPick two servers at random, send to the less busyNearly as good as least connections; no global state, so many LBs can run it independentlySlightly less even than a perfect choiceMany load balancer instances at large scale
Hash (IP or key)Hash the client IP or a key to a serverSame client or key always reaches the same server (useful for caches)Uneven if a few keys are hot; IPs change and many users share one office IPCache affinity, stateful protocols; use consistent hashing so adding a server moves few keys
import random

def pick(servers):
    a, b = random.sample(servers, 2)          # two different servers at random
    return a if a.active_requests <= b.active_requests else b

Here is a minimal layer 7 setup in Nginx, showing a pool, an algorithm, a weight and passive health checks (open-source Nginx only marks a server down after real requests fail; active probes need Nginx Plus or another proxy):

upstream shop_api {
    least_conn;                                    # pick the server with fewest active requests
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.13:8080 weight=2;                # a bigger machine takes twice the share
    keepalive 64;                                  # reuse connections to the backends
}

server {
    listen 443 ssl;
    server_name api.shop.com;
    ssl_certificate     /etc/ssl/shop.crt;
    ssl_certificate_key /etc/ssl/shop.key;

    location / {
        proxy_pass http://shop_api;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_next_upstream error timeout;         # try another server if one fails to answer
    }
}

Health checks

A server stops answering and is taken out of rotation

  1. Load balancers also run health checks. Every few seconds they ping each server. If one stops answering, it is taken out of rotation, so after a few seconds no new requests are sent to it.

There are two kinds, and good setups use both:

  • Active health checks: the LB calls an endpoint such as GET /healthz on every backend every few seconds. After a number of failures in a row (the unhealthy threshold, often 2–3) the server is removed; after several successes it is added back.
  • Passive health checks (outlier detection): the LB watches real traffic, and a server returning errors or timing out is ejected for a while.

Detection is not instant. With a 5-second interval and a threshold of 3, a dead server can keep receiving requests for up to about 15 seconds. Requests already in flight on a crashed server fail too. Clients and the LB should retry idempotent requests (ones that are safe to repeat, like a GET) on another server; a payment should only be retried with an idempotency key so it cannot be charged twice.

Stateless servers: the rule that makes scaling out work

A load balancer can send a request anywhere only if any server can handle any request, so app servers must be stateless: they keep nothing between requests that another server would need. Anything that must survive goes to a shared store.

The classic trap is the session: the record that a user is logged in, or what is in their basket. If it lives in one server's memory, the next request lands on a different server and the user is suddenly logged out.

OptionHowGood atWeak at
Sticky sessionsLB sends a user back to the same server, usually by a cookie it setsNo code changeUneven load; when that server dies its users lose their sessions; deploys log people out
Shared session storeSessions in Redis (or a database), keyed by a session ID in a cookieAny server can serve anyone; sessions survive server lossOne more system on the request path (~0.5 ms); it must be replicated
Signed tokens (e.g. JWT)The session data itself travels in a token the server signs and verifiesNo lookup at all; scales triviallyHard to revoke before expiry; tokens grow; keep them short-lived with a refresh step

The same rule applies to everything else a server might keep locally: uploaded files go to object storage, caches that must agree go to a shared cache, scheduled jobs must not run once per server. Once servers are stateless they become interchangeable, and an autoscaler can add or remove them as load changes.

Worked example: how many servers?

Need three, run four

  1. Put it together: scale out behind a load balancer, and always keep one spare server, called N plus one, so a single crash never takes you down. That is how real sites serve millions of people.

Your API peaks at 9,000 requests per second. Load testing shows one server saturates at about 1,000 per second, but latency climbs steeply past ~70% busy (queues build up; see latency vs throughput), so you plan for 700 per second each.

  1. Bare minimum: 9,000 ÷ 700 ≈ 12.9, so 13 servers.
  2. N+1: one spare so a single failure (or a server out for deploy) still leaves enough: 14 servers.
  3. Survive losing a whole data centre zone. Clouds group data centres into availability zones that fail independently. Spread across 3 zones, losing one leaves 2, and those 2 must carry the full 9,000: 4,500 per zone ÷ 700 ≈ 6.4, so 7 per zone, 21 servers.

The jump from 14 to 21 is the real price of surviving a zone failure, and stating it is exactly the sort of detail interviewers want.

Your turn: the concert ticket sale

A ticket site normally sees 2,000 requests per second. When a big concert goes on sale, it gets 40,000 per second for about 15 minutes. One app server handles 800 per second at your target utilisation. The site runs in 2 availability zones. Work out how many servers you need for the sale, what else in the design changes, and whether autoscaling alone is enough.

Compare with our answer
  • Sale capacity: 40,000 ÷ 800 = 50 servers. To survive losing one of 2 zones, each zone must carry everything: 50 per zone, 100 servers, or accept a degraded sale if a zone fails.
  • Normal days: 2,000 ÷ 800 = 3, so 3–4 per zone. The sale needs 25× more.
  • Autoscaling alone is too slow: new servers take minutes to boot and warm up, and the spike arrives in seconds. Scale up before the sale starts (scheduled scaling), then let autoscaling trim afterwards.
  • The app tier is the easy part. The database and the seat inventory are the real limits: 40,000 requests per second fighting over the same seats. Put a virtual waiting room (a queue that admits users at a controlled rate) in front.
  • The load balancer must handle the spike too: pre-warm a managed LB or use a layer 4 tier in front, and use least connections or power of two choices because some requests (checkout) are much slower than others.

A strong answer mentions: servers = peak ÷ per-server capacity · zone redundancy doubles it with 2 zones · scale before the spike, not during it · the database or inventory is the real bottleneck · a waiting room to admit users gradually.

Options and trade-offs: up or out?

Scale up (vertical)Scale out (horizontal)
Good atNo code changes; no distributed problems; one place for data; strong per-machine performanceNear-unlimited capacity; survives a machine failure; add or remove capacity in minutes
Weak atHard ceiling; single point of failure; resize usually needs a restart; price premium at the top endNeeds a load balancer and stateless servers; more machines to run and monitor; harder for stateful systems
Use whenDatabases and other stateful systems, first; anything that fits on one machine with headroomStateless app and web tiers, always; when one machine is not enough or one failure is unacceptable

In practice you do both: app servers scale out behind a load balancer from day one, while the database scales up as far as is sensible before you add replicas and then shards.

Other ways it goes wrong

  • The load balancer is a single point of failure too. Run it as a pair (one active, one standby that takes over a shared "floating" IP address), or use a managed LB that runs on many nodes behind the scenes. DNS can list several LB addresses.
  • Cascading failure. One of 10 servers dies and its traffic moves to the other 9, each now 11% busier. If they were already at 90%, they tip over one after another. Headroom (N+1 and the 70% target) and load shedding (rejecting excess requests quickly with a 503) stop the domino effect.
  • Retry storms. If every layer retries 3 times, one slow request can turn into 27 at the bottom. Retry only at one layer, with exponential backoff, jitter and a retry budget (circuit breakers and retries).
  • Long-lived connections don't rebalance. WebSocket clients stay on the server they first reached, so a newly added server gets no traffic until clients reconnect. Servers can ask clients to reconnect gradually to spread out.

When not to scale out (yet)

  • One machine with headroom is enough. If peak load is 20% of one server, a second server is for availability, not capacity: two is the right number, not twenty.
  • The bottleneck is the database. Adding app servers in front of a saturated database sends it even more queries. Fix the database first: indexes, a cache, read replicas.
  • The work is stateful and hard to split. A database, a game world, a single queue with strict ordering. Scale these up first; scaling them out means sharding, which is a big design change.

In the interview

Your turn: spot the bug

A team adds a second app server and a round robin load balancer. Within an hour: users are randomly logged out, some uploaded profile photos show as broken images, and the "send daily digest email" job sends every email twice. Explain each symptom and fix them.

Compare with our answer
  • Logouts: sessions are kept in each server's memory, so a request landing on the other server finds no session. Move sessions to a shared store (Redis) or use signed tokens; sticky sessions only hide it.
  • Broken photos: uploads are saved to the local disk of whichever server received them, so the other server can't find them. Store files in object storage and serve them through a CDN.
  • Double emails: the scheduled job runs on every server. Run it in one place (a separate worker or scheduler), or have servers take a lock or lease before running it, and make the send idempotent so a repeat does no harm.
  • The general rule: app servers must be stateless. Anything that must persist or happen once goes to a shared system.

A strong answer mentions: in-memory sessions · local disk uploads · jobs that run on every server · moving all three out to shared systems · why sticky sessions are not the fix.

Scaling and load balancing on one page
  • Scale up = bigger machine: simple, but a ceiling, a single point of failure and a restart to resize. Do it first for databases.
  • Scale out = more machines behind a load balancer: near-unlimited and survives failures, but servers must be stateless.
  • L4 balances TCP connections (fast, any protocol, WebSockets); L7 reads HTTP (path routing, TLS termination, retries).
  • Algorithms: round robin (identical work) · weighted (mixed sizes) · least connections / power of two choices (mixed request cost) · consistent hashing (affinity).
  • Health checks: active probes + passive outlier detection; detection takes interval × threshold (5 s × 3 ≈ 15 s); drain connections on deploy.
  • Sessions: shared store (Redis) or signed tokens; sticky sessions are a stopgap.
  • Sizing: peak ÷ (capacity at ~70%) → +1 spare → enough to lose a whole zone (3 zones: each pair must carry the peak).
  • Breaks: least-connections black hole from a fast-failing server; deep health checks removing every server; cascading overload; retry storms; the LB itself as a single point of failure.
Interactive labPractise it: balance a lunch rush, survive a crash and handle a mixed fleetOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. Your single Postgres primary runs at 70% CPU on 16 cores, mostly writes, and traffic grows about 30% a year. App servers are fine. What is the most sensible next step?

2. After adding a second app server behind a round robin load balancer, users are randomly logged out. What is the best fix?

3. Your pool has two 8-core servers and one 16-core server, and every request costs about the same. Which algorithm spreads the load best?

4. After a deploy, one server returns an error for every request within 2 ms. The load balancer uses least connections, and the error rate climbs to 60%. Why?

5. Your health endpoint runs a database query. The database fails over for 20 seconds and the whole site goes down, even pages that don't use the database. What is the best change?

6. You need to send /api/* to one pool and /video/* to another, and terminate TLS at the edge. Which load balancer fits?

7. Peak traffic is 6,000 requests per second, one server handles 500 at your target utilisation, and the service runs in 3 availability zones that must survive losing any one zone. How many servers do you run?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.