Scaling and load balancing
Vertical vs horizontal scaling, how load balancers spread traffic (algorithms, L4 vs L7, health checks), stateless servers and sessions, and sizing a fleet.
On this page
The problem: one server, and the traffic keeps growing
Your shop's API runs on one server. Load-tested, it handles about 200 requests per second before it slows to a crawl. On a normal day you get 50. Then a marketing email goes out and 300 requests per second arrive. The server's processors are at 100%, requests queue up, time out and fail. Customers see errors at exactly the moment you paid to attract them.
One server, three times more traffic than it can take
- A server can only handle so many requests each second. Ours handles about 200. As more people show up, it gets busier and busier, until it simply cannot keep up.
There are only two ways to get more capacity: make the server bigger, or add more servers. Each has a different cost, and the second needs a new component, the load balancer, plus a change to how you write your application.
The idea: scale up, or scale out
Scaling up works, until it doesn't
- Option one: scale up. Swap it for a bigger, more powerful machine, with more processors and more memory. Now it handles 400 requests a second.
- Scaling up is simple, but it has limits. Bigger machines get expensive fast, there is a largest machine you can buy, and it is still just one machine. If it breaks, everything stops.
Scaling up (vertical scaling) means moving to a bigger machine: more processor cores, more memory, faster disks. Nothing in your code changes. Cloud providers rent machines with hundreds of cores and several terabytes of memory, so the ceiling is high. But:
- There is still a ceiling, and the very largest machines cost more per unit of capacity than mid-sized ones.
- It is still one machine. When it crashes or is patched, everything stops. It is a single point of failure: one part whose failure takes the whole system down.
- Resizing usually means a restart, which is downtime unless you have a second machine to fail over to.
- Software doesn't always use the extra cores. A lock that every request needs, or a single-threaded step, caps the speed-up no matter how many cores you add.
Scaling out: more ordinary servers, but who sends them traffic?
- Option two: scale out. Instead of one huge machine, use several ordinary ones. Need more capacity? Just add another server.
- But users only know one address. Without help, every request still goes to the same server, which is overloaded, while the other two sit idle.
Scaling out (horizontal scaling) means running several ordinary machines side by side and sharing the work between them. Capacity grows by adding machines, and losing one only loses a share of capacity. But users know one address, so something must stand in front and divide the requests.
How it works: the load balancer
The load balancer receives every request and passes it on
- Enter the load balancer. It sits in front of the servers, receives every request, and passes each one on to a server that can take it.
A load balancer (LB) is a server (or managed service) that receives every incoming request and forwards it to one of a pool of backend servers. Clients only ever see the load balancer's address, usually through a DNS name such as api.shop.com. The backends can be added, removed and replaced without clients knowing.
A load balancer typically also:
- Terminates TLS: it does the HTTPS encryption work once, so backends receive plain HTTP inside the private network.
- Health-checks backends and stops sending traffic to broken ones.
- Drains connections during deploys: a server being removed gets no new requests but is allowed to finish the ones it has (often 30–300 s).
Layer 4 or layer 7?
The "layer" refers to how much of the request the balancer reads. Layer 4 (the transport layer) sees only IP addresses and ports of a TCP or UDP connection. Layer 7 (the application layer) reads the HTTP request itself: the path, headers and cookies.
| Layer 4 (TCP/UDP) | Layer 7 (HTTP) | |
|---|---|---|
| Decides by | IP address and port, once per connection | Each request's path, host, headers, cookies |
| Good at | Very high throughput, low added latency (microseconds); any protocol; long-lived connections such as WebSockets or database traffic | Routing /api and /images to different pools; TLS termination; retries; per-request balancing over one connection; adding headers |
| Weak at | Can't route by URL or retry a failed HTTP request; one heavy connection stays on one server | More CPU per request; adds roughly a millisecond; must understand the protocol |
| Examples | AWS Network Load Balancer, Google Maglev, Linux IPVS, HAProxy in TCP mode | Nginx, Envoy, HAProxy in HTTP mode, AWS Application Load Balancer |
Large sites often use both: a layer 4 tier spreads raw connections across a fleet of layer 7 proxies, which then route each HTTP request to the right service.
Choosing a server: the algorithms
Taking turns, or picking the least busy
- How does it choose? Round robin simply takes turns: one, two, three, one, two, three. Least connections picks whichever server is least busy right now, which works better when some requests take longer.
| Algorithm | How it picks | Good at | Weak at | Use when |
|---|---|---|---|---|
| Round robin | Takes turns: 1, 2, 3, 1, 2, 3 | Simple, even spread | Ignores server size and request cost | Identical servers, similar requests |
| Weighted round robin | Turns in proportion to a weight (a 2× machine gets 2 shares) | Mixed server sizes | Weights are static; slow requests still pile up | Known, stable differences in capacity |
| Least connections | Server with the fewest requests in progress | Requests of very different lengths | Needs a live count per server; can be fooled (see below) | Mixed fast and slow requests |
| Power of two choices | Pick two servers at random, send to the less busy | Nearly as good as least connections; no global state, so many LBs can run it independently | Slightly less even than a perfect choice | Many load balancer instances at large scale |
| Hash (IP or key) | Hash the client IP or a key to a server | Same client or key always reaches the same server (useful for caches) | Uneven if a few keys are hot; IPs change and many users share one office IP | Cache affinity, stateful protocols; use consistent hashing so adding a server moves few keys |
import random
def pick(servers):
a, b = random.sample(servers, 2) # two different servers at random
return a if a.active_requests <= b.active_requests else b
Here is a minimal layer 7 setup in Nginx, showing a pool, an algorithm, a weight and passive health checks (open-source Nginx only marks a server down after real requests fail; active probes need Nginx Plus or another proxy):
upstream shop_api {
least_conn; # pick the server with fewest active requests
server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
server 10.0.1.13:8080 weight=2; # a bigger machine takes twice the share
keepalive 64; # reuse connections to the backends
}
server {
listen 443 ssl;
server_name api.shop.com;
ssl_certificate /etc/ssl/shop.crt;
ssl_certificate_key /etc/ssl/shop.key;
location / {
proxy_pass http://shop_api;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_next_upstream error timeout; # try another server if one fails to answer
}
}
Health checks
A server stops answering and is taken out of rotation
- Load balancers also run health checks. Every few seconds they ping each server. If one stops answering, it is taken out of rotation, so after a few seconds no new requests are sent to it.
There are two kinds, and good setups use both:
- Active health checks: the LB calls an endpoint such as
GET /healthzon every backend every few seconds. After a number of failures in a row (the unhealthy threshold, often 2–3) the server is removed; after several successes it is added back. - Passive health checks (outlier detection): the LB watches real traffic, and a server returning errors or timing out is ejected for a while.
Detection is not instant. With a 5-second interval and a threshold of 3, a dead server can keep receiving requests for up to about 15 seconds. Requests already in flight on a crashed server fail too. Clients and the LB should retry idempotent requests (ones that are safe to repeat, like a GET) on another server; a payment should only be retried with an idempotency key so it cannot be charged twice.
Stateless servers: the rule that makes scaling out work
A load balancer can send a request anywhere only if any server can handle any request, so app servers must be stateless: they keep nothing between requests that another server would need. Anything that must survive goes to a shared store.
The classic trap is the session: the record that a user is logged in, or what is in their basket. If it lives in one server's memory, the next request lands on a different server and the user is suddenly logged out.
| Option | How | Good at | Weak at |
|---|---|---|---|
| Sticky sessions | LB sends a user back to the same server, usually by a cookie it sets | No code change | Uneven load; when that server dies its users lose their sessions; deploys log people out |
| Shared session store | Sessions in Redis (or a database), keyed by a session ID in a cookie | Any server can serve anyone; sessions survive server loss | One more system on the request path (~0.5 ms); it must be replicated |
| Signed tokens (e.g. JWT) | The session data itself travels in a token the server signs and verifies | No lookup at all; scales trivially | Hard to revoke before expiry; tokens grow; keep them short-lived with a refresh step |
The same rule applies to everything else a server might keep locally: uploaded files go to object storage, caches that must agree go to a shared cache, scheduled jobs must not run once per server. Once servers are stateless they become interchangeable, and an autoscaler can add or remove them as load changes.
Worked example: how many servers?
Need three, run four
- Put it together: scale out behind a load balancer, and always keep one spare server, called N plus one, so a single crash never takes you down. That is how real sites serve millions of people.
Your API peaks at 9,000 requests per second. Load testing shows one server saturates at about 1,000 per second, but latency climbs steeply past ~70% busy (queues build up; see latency vs throughput), so you plan for 700 per second each.
- Bare minimum: 9,000 ÷ 700 ≈ 12.9, so 13 servers.
- N+1: one spare so a single failure (or a server out for deploy) still leaves enough: 14 servers.
- Survive losing a whole data centre zone. Clouds group data centres into availability zones that fail independently. Spread across 3 zones, losing one leaves 2, and those 2 must carry the full 9,000: 4,500 per zone ÷ 700 ≈ 6.4, so 7 per zone, 21 servers.
The jump from 14 to 21 is the real price of surviving a zone failure, and stating it is exactly the sort of detail interviewers want.
A ticket site normally sees 2,000 requests per second. When a big concert goes on sale, it gets 40,000 per second for about 15 minutes. One app server handles 800 per second at your target utilisation. The site runs in 2 availability zones. Work out how many servers you need for the sale, what else in the design changes, and whether autoscaling alone is enough.
Compare with our answer
- Sale capacity: 40,000 ÷ 800 = 50 servers. To survive losing one of 2 zones, each zone must carry everything: 50 per zone, 100 servers, or accept a degraded sale if a zone fails.
- Normal days: 2,000 ÷ 800 = 3, so 3–4 per zone. The sale needs 25× more.
- Autoscaling alone is too slow: new servers take minutes to boot and warm up, and the spike arrives in seconds. Scale up before the sale starts (scheduled scaling), then let autoscaling trim afterwards.
- The app tier is the easy part. The database and the seat inventory are the real limits: 40,000 requests per second fighting over the same seats. Put a virtual waiting room (a queue that admits users at a controlled rate) in front.
- The load balancer must handle the spike too: pre-warm a managed LB or use a layer 4 tier in front, and use least connections or power of two choices because some requests (checkout) are much slower than others.
A strong answer mentions: servers = peak ÷ per-server capacity · zone redundancy doubles it with 2 zones · scale before the spike, not during it · the database or inventory is the real bottleneck · a waiting room to admit users gradually.
Options and trade-offs: up or out?
| Scale up (vertical) | Scale out (horizontal) | |
|---|---|---|
| Good at | No code changes; no distributed problems; one place for data; strong per-machine performance | Near-unlimited capacity; survives a machine failure; add or remove capacity in minutes |
| Weak at | Hard ceiling; single point of failure; resize usually needs a restart; price premium at the top end | Needs a load balancer and stateless servers; more machines to run and monitor; harder for stateful systems |
| Use when | Databases and other stateful systems, first; anything that fits on one machine with headroom | Stateless app and web tiers, always; when one machine is not enough or one failure is unacceptable |
In practice you do both: app servers scale out behind a load balancer from day one, while the database scales up as far as is sensible before you add replicas and then shards.
Other ways it goes wrong
- The load balancer is a single point of failure too. Run it as a pair (one active, one standby that takes over a shared "floating" IP address), or use a managed LB that runs on many nodes behind the scenes. DNS can list several LB addresses.
- Cascading failure. One of 10 servers dies and its traffic moves to the other 9, each now 11% busier. If they were already at 90%, they tip over one after another. Headroom (N+1 and the 70% target) and load shedding (rejecting excess requests quickly with a 503) stop the domino effect.
- Retry storms. If every layer retries 3 times, one slow request can turn into 27 at the bottom. Retry only at one layer, with exponential backoff, jitter and a retry budget (circuit breakers and retries).
- Long-lived connections don't rebalance. WebSocket clients stay on the server they first reached, so a newly added server gets no traffic until clients reconnect. Servers can ask clients to reconnect gradually to spread out.
When not to scale out (yet)
- One machine with headroom is enough. If peak load is 20% of one server, a second server is for availability, not capacity: two is the right number, not twenty.
- The bottleneck is the database. Adding app servers in front of a saturated database sends it even more queries. Fix the database first: indexes, a cache, read replicas.
- The work is stateful and hard to split. A database, a game world, a single queue with strict ordering. Scale these up first; scaling them out means sharding, which is a big design change.
In the interview
A team adds a second app server and a round robin load balancer. Within an hour: users are randomly logged out, some uploaded profile photos show as broken images, and the "send daily digest email" job sends every email twice. Explain each symptom and fix them.
Compare with our answer
- Logouts: sessions are kept in each server's memory, so a request landing on the other server finds no session. Move sessions to a shared store (Redis) or use signed tokens; sticky sessions only hide it.
- Broken photos: uploads are saved to the local disk of whichever server received them, so the other server can't find them. Store files in object storage and serve them through a CDN.
- Double emails: the scheduled job runs on every server. Run it in one place (a separate worker or scheduler), or have servers take a lock or lease before running it, and make the send idempotent so a repeat does no harm.
- The general rule: app servers must be stateless. Anything that must persist or happen once goes to a shared system.
A strong answer mentions: in-memory sessions · local disk uploads · jobs that run on every server · moving all three out to shared systems · why sticky sessions are not the fix.
- Scale up = bigger machine: simple, but a ceiling, a single point of failure and a restart to resize. Do it first for databases.
- Scale out = more machines behind a load balancer: near-unlimited and survives failures, but servers must be stateless.
- L4 balances TCP connections (fast, any protocol, WebSockets); L7 reads HTTP (path routing, TLS termination, retries).
- Algorithms: round robin (identical work) · weighted (mixed sizes) · least connections / power of two choices (mixed request cost) · consistent hashing (affinity).
- Health checks: active probes + passive outlier detection; detection takes interval × threshold (5 s × 3 ≈ 15 s); drain connections on deploy.
- Sessions: shared store (Redis) or signed tokens; sticky sessions are a stopgap.
- Sizing: peak ÷ (capacity at ~70%) → +1 spare → enough to lose a whole zone (3 zones: each pair must carry the peak).
- Breaks: least-connections black hole from a fast-failing server; deep health checks removing every server; cascading overload; retry storms; the LB itself as a single point of failure.
1. Your single Postgres primary runs at 70% CPU on 16 cores, mostly writes, and traffic grows about 30% a year. App servers are fine. What is the most sensible next step?
A stateful database is the classic case for scaling up first: it is simple and buys a few years. Sharding is a large change that isn't needed yet; more app servers or a load balancer can't spread writes that all go to one primary.
2. After adding a second app server behind a round robin load balancer, users are randomly logged out. What is the best fix?
Sessions live in one server's memory, so requests that land elsewhere find nothing. Moving them to shared storage (or into signed tokens) makes servers stateless. Sticky sessions hide the problem until a server restarts.
3. Your pool has two 8-core servers and one 16-core server, and every request costs about the same. Which algorithm spreads the load best?
With equal requests and known capacities, weights are the direct fix. Plain round robin and random give the small servers the same share as the big one; IP hash depends on how clients are spread, not on capacity.
4. After a deploy, one server returns an error for every request within 2 ms. The load balancer uses least connections, and the error rate climbs to 60%. Why?
A server that fails instantly always looks least busy, so it attracts traffic: a "black hole". Passive health checks that eject servers with high error rates, plus slow start for new servers, prevent it.
5. Your health endpoint runs a database query. The database fails over for 20 seconds and the whole site goes down, even pages that don't use the database. What is the best change?
A shared dependency failing made every server look broken at once, so the LB removed them all. Health checks should test what is specific to each server, and failing open avoids an empty pool. A 60-second interval would also delay removing truly dead servers.
6. You need to send /api/* to one pool and /video/* to another, and terminate TLS at the edge. Which load balancer fits?
Routing by URL path requires reading the HTTP request, which only a layer 7 balancer does. Layer 4 sees only addresses and ports, and DNS cannot see paths at all.
7. Peak traffic is 6,000 requests per second, one server handles 500 at your target utilisation, and the service runs in 3 availability zones that must survive losing any one zone. How many servers do you run?
With one zone gone, the remaining 2 must carry all 6,000: 3,000 per zone ÷ 500 = 6 per zone, 18 in total. 12 is the bare minimum with no spare and 13 is N+1, but neither survives losing a third of the fleet.
Score 60% or more and the lesson is marked complete.