ZooKeeper and etcd
How coordination services work: consensus with ZAB and Raft, leader election, locks with fencing tokens, ephemeral nodes and watches, etcd today, and why Redis locks need fencing.
On this page
The problem: machines that must agree
A billing service runs on three servers so that one can fail. Exactly one of them must run the nightly invoice job; if two run it, customers are charged twice. A database cluster must agree on which node is the primary. A fleet of workers needs to know which workers are alive and which config version is current.
The obvious solutions fail in the usual ways. A row in a database saying "server B is leader" is a single point of failure, and nothing clears it when server B dies. A heartbeat between the three servers breaks during a network cut: each side thinks the other is dead and both become leader (split brain). Clocks drift, processes pause, and messages arrive late.
Coordination services solve this once, carefully, so every other system can rely on them. Apache ZooKeeper (from Yahoo, around 2008) and etcd (from CoreOS, 2013, now the store behind every Kubernetes cluster) are the two you should know. They are small, strongly consistent, highly available stores for small amounts of critical data: who is leader, who holds a lock, which servers are members, what the config says. They are not for application data.
Consensus, intuitively
Both services run on an odd number of servers (an ensemble or cluster: 3 or 5) and use a consensus protocol so that all servers agree on one ordered log of changes, even if some crash. ZooKeeper uses ZAB (ZooKeeper Atomic Broadcast); etcd uses Raft. They differ in detail but share the same core ideas:
- One server is the leader; all writes go through it, which gives a single order.
- A write is committed once a majority (a quorum) of servers has stored it on disk. Then it can never be lost, because any future majority overlaps with this one in at least one server.
- If the leader dies, the others hold an election. Each new leader has a higher term (Raft) or epoch (ZAB), and a server only votes for a candidate whose log is at least as up to date as its own, so the winner has every committed write.
- Messages from a leader with an old term are rejected, which stops a deposed leader doing damage.
Writes commit on a majority; a minority side cannot commit anything
That is why ensembles have odd sizes: 3 servers tolerate 1 failure, 5 tolerate 2, and a 4th or 6th server adds no extra tolerance (4 still needs 3 for a majority). More servers make writes slower, because each needs more acknowledgements, so 3 or 5 is normal and 7 is rare. The same idea underlies split-brain prevention in replication.
ZooKeeper's data model: znodes, sessions and watches
ZooKeeper stores a tree of znodes, like a file system where every node can hold a small amount of data (1 MB at most by default; in practice keep it to a few KB) and children. Each znode has a version, and writes can be conditional on it ("set if version is still 7"), which is an atomic compare-and-set.
Three features make coordination recipes possible:
- Sessions and ephemeral znodes. A client holds a session, kept alive by heartbeats. An ephemeral znode is deleted automatically when the session that created it ends, whether the client closed it or simply stopped heartbeating for the session timeout (typically a few seconds to tens of seconds). "This node exists" therefore means "its owner is alive".
- Sequential znodes. ZooKeeper appends an increasing counter to the name: create
/election/n_and get/election/n_0000000007. This gives a fair queue. - Watches. A client can ask to be notified when a znode changes or its children change. Classic watches fire once and must be set again; ZooKeeper 3.6 added persistent and recursive watches.
Consistency: writes are linearizable (one total order, as if on one machine). Reads are served by whichever server the client is connected to, so they can be slightly behind the leader, but each client sees changes in order and never goes back in time. Call sync before a read when you need the very latest value.
The recipes
Leader election
Ephemeral sequential nodes: the lowest is leader; each watches only its predecessor
Every candidate creates an ephemeral sequential node; the lowest number leads. Watching only the predecessor (instead of everyone watching the leader) avoids the herd effect, where one change wakes hundreds of clients at once. If a node in the middle dies, only its successor re-checks and starts watching the next one down.
Membership and configuration
- Group membership: each worker creates an ephemeral node
/workers/w-17; anyone can list the children of/workersto see who is alive and watch it for changes. Kafka (before KRaft), HBase and Solr have used ZooKeeper exactly this way. - Configuration: store
/config/feature-flagsand have services watch it. Updates reach all watchers within moments, and the version number tells them which config they have.
Locks
A lock is the election recipe with a different name: create /locks/invoice-run/n_, and you hold the lock when your node is the lowest. Release by deleting it; a crash releases it automatically when the session expires. That expiry is the catch.
Locks need fencing tokens
Every distributed lock is really a lease: it is held for as long as the holder stays alive in the eyes of the lock service. A holder can lose its lease without knowing it: a garbage-collection pause of several seconds, a VM migration, a slow disk, or a network cut long enough for the session to expire. When it resumes, it carries on as if it still held the lock.
A paused client wakes with an old token, and the storage rejects its write
The fix is a fencing token: a number that increases every time the lock is granted, sent with every write to the protected resource, which rejects any token lower than the highest it has seen. Both services provide one: in ZooKeeper, the zxid (transaction ID) or the sequence number of the lock node; in etcd, the revision at which the lock was acquired.
func runInvoices(ctx context.Context, cli *clientv3.Client, store *Store) error {
sess, err := concurrency.NewSession(cli, concurrency.WithTTL(10))
if err != nil {
return err
}
defer sess.Close()
m := concurrency.NewMutex(sess, "/locks/invoice-run")
if err := m.Lock(ctx); err != nil {
return err
}
defer m.Unlock(ctx)
token := m.Header().Revision // grows with every acquisition: the fencing token
return store.RunIfNewer(ctx, "invoice-run", token)
}
The resource must check it, for example in SQL:
UPDATE job_state SET fence = :token, started_at = now()
WHERE job = 'invoice-run' AND fence < :token; -- 0 rows updated = you lost the lock
etcd: the modern alternative
etcd is a key-value store run on Raft, written in Go, with a gRPC API. The concepts map closely onto ZooKeeper's:
| Need | ZooKeeper | etcd |
|---|---|---|
| Data model | Tree of znodes | Flat keys; "directories" are key prefixes queried as ranges |
| Liveness | Session + ephemeral znode | Lease with a TTL, kept alive by the client; keys attached to it vanish when it expires |
| Change notification | Watches (one-shot; persistent since 3.6) | Watch streams on a key or prefix, starting from any past revision, so no change is missed between reconnects |
| Atomic multi-step | multi (a batch of operations, all or nothing) | Transactions: if (compare) then (ops) else (ops) |
| Versions | Per-znode version, global zxid | Every change bumps a global revision; MVCC keeps history until compacted |
| Reads | Local server (may lag); sync to catch up | Linearizable by default (the leader confirms it is still leader); serializable reads optional for speed |
etcd's watch-from-revision is a real improvement: with one-shot watches, a change between a notification and setting the next watch could be missed unless you re-read; with etcd you resume from the last revision you saw. That is how Kubernetes controllers keep up with cluster state.
Which one? For a new system, etcd is the usual default, especially near Kubernetes. ZooKeeper remains in many existing systems (HBase, Solr, Hadoop, older Kafka). Kafka itself replaced ZooKeeper with its own Raft-based controllers (KRaft) and removed ZooKeeper support in Kafka 4.0, the same move many systems have made to avoid running a separate ensemble. Consul (also Raft) is a third option, popular for service discovery.
Performance and limits
| Measure | Typical value |
|---|---|
| Write latency | a few ms in one region, bounded by disk fsync and a majority round trip |
| Write throughput | thousands to a few tens of thousands of writes/s per cluster; it does not grow with more servers |
| Reads | higher; ZooKeeper reads scale with servers (and observers), etcd linearizable reads go through the leader check |
| Data size | ZooKeeper: everything in memory, a few GB at most; etcd: 2 GB default quota, ~8 GB recommended maximum, 1.5 MiB per request |
| Failover | a new leader within roughly 1–10 s, depending on timeouts |
Worked example. 2,000 workers each renew a lease every 3 seconds (≈ 670 writes/s), plus 50 leader elections and config changes a minute: comfortably within one 3-node etcd cluster. But 50,000 jobs a second each taking and releasing a lock is 100,000 writes/s, beyond any single consensus cluster: partition the work (Kafka partitions or sharding give each worker exclusive keys) instead of locking per job.
When to use a coordination service, and when not
| ZooKeeper / etcd | Redis lock | Database row lock or constraint | Kafka partitions | |
|---|---|---|---|---|
| Good at | Leader election, membership, config, correct locks with fencing | Fast, cheap, best-effort mutual exclusion | Exclusivity for data in that database | Exclusive ownership of a key range by one consumer |
| Weak at | High write volume, large data | Correctness across failover; no fencing | Coordination outside that database | Fine-grained, ad-hoc locks |
| Use when | A few critical decisions must be agreed | Duplicate work is harmless | The protected data lives in that database | Work can be divided by key |
When not to use one: as a general database or queue (both are small and write-limited); for per-request locks at high volume; when the protected data already lives in a database that can enforce the rule itself (SELECT … FOR UPDATE, a unique index); and when your platform already provides leader election (Kubernetes leases, a managed database's failover), which is usually built on one of these anyway.
In the interview
A cron-like scheduler runs on three servers for availability. Every minute, the active one reads due jobs from PostgreSQL and enqueues them. A job must never be enqueued twice. Design the leader election, what happens when the leader dies or pauses, and how duplicate enqueues are prevented.
Compare with our answer
Run a 3-node etcd (or use the platform's lease API). Each scheduler campaigns for /scheduler/leader with a lease of ~10 s, renewed every ~3 s; the winner gets the election revision as its fencing token. When the leader dies, its lease expires within ~10 s and another instance wins; schedules a minute apart tolerate that gap. To protect against a paused leader, enqueueing is guarded twice: the scheduler claims each run with UPDATE runs SET claimed_by_token = :token WHERE job_id = ? AND due_at = ? AND claimed_by_token < :token, or inserts into a table with a unique constraint on (job_id, due_at), so a second enqueue of the same run fails whoever does it. Workers also treat jobs idempotently.
A strong answer mentions: election with a lease and its TTL · the fencing token from the coordination service · the resource rejecting old tokens · a unique constraint or idempotency as a second guard · the failover gap and why it is acceptable · etcd's 3-node majority.
A team stores per-user shopping carts in ZooKeeper "because it is strongly consistent", one znode per user, 10 million users. Each service instance sets a watch on /config and on the leader's znode; on every change, all 800 instances re-read everything. Their lock is a single znode /lock: whoever creates it holds it, everyone else watches it. The ensemble has 4 servers "for extra safety". What would you change?
Compare with our answer
A strong answer mentions:
- Carts are application data: 10 M znodes in memory and constant writes overload a store meant for small coordination data; carts belong in a database or Redis.
- A single
/locknode that everyone watches causes a herd effect: each release wakes all waiters, who all race to create it; use ephemeral sequential nodes with each waiter watching its predecessor. - The lock has no fencing token, so a paused holder can still write after its session expires; pass the
zxidor sequence number and check it at the resource. - 4 servers tolerate only 1 failure (like 3) but add write latency; use 3 or 5.
- 800 instances re-reading everything on each change is a burst of reads on every update; read only the changed node, or use etcd watches from a revision.
- Small, strongly consistent, highly available stores for coordination data: leaders, locks, membership, config. Not a database.
- Consensus (ZAB in ZooKeeper, Raft in etcd): one leader orders writes; commit on a majority; elections with increasing terms/epochs.
- 3 nodes tolerate 1 failure, 5 tolerate 2; even sizes add nothing; more nodes = slower writes.
- ZooKeeper: znode tree, versions for compare-and-set, ephemeral nodes tied to sessions, sequential nodes, watches (one-shot; persistent since 3.6). Reads may lag;
syncto catch up. - etcd: flat keys, leases, watch streams from a revision,
if/then/elsetransactions, linearizable reads by default. Kubernetes runs on it. - Election and locks: lowest ephemeral sequential node wins; watch only the predecessor (no herd).
- Every lock is a lease: a paused holder can lose it unknowingly. Fencing tokens (zxid, etcd revision), checked by the resource, make it safe.
- Redis locks: fine for efficiency; for correctness use consensus + fencing, or a database constraint.
- Throughput: thousands to tens of thousands of writes/s; keep it off the per-request path.
1. A team runs a 4-node ZooKeeper ensemble instead of 3 "to survive more failures". How many servers can fail while writes still succeed?
Writes need a majority. With 4 servers that is 3, so only 1 may fail, as with 3 servers, while every write waits for more acknowledgements. Use 3 or 5.
2. A worker holding a ZooKeeper lock pauses for 30 seconds in garbage collection. Its session times out, another worker takes the lock, and then the first worker resumes. What prevents both writing to the database?
After resuming, the first worker may write before it learns its session is gone. Only the resource checking an increasing token rejects the stale write; no timeout bounds every pause.
3. 500 clients wait for one lock, and every client watches the single lock znode. What happens each time the lock is released?
Everyone watching one node is woken together. With ephemeral sequential nodes, each client watches only its predecessor, so one release wakes one client.
4. Why is a lock built with SET key token NX PX 10000 on a single Redis primary with one replica unsafe for correctness?
The key may not reach the replica before the primary fails, and there is no fencing token. SET … NX PX is atomic, Redis runs commands one at a time, and the expiry uses the server's clock.
5. A network cut splits a 5-node etcd cluster into groups of 2 (including the old leader) and 3. What happens?
Only a majority (3 of 5) can commit, so the minority side, even with the old leader, is stuck. Requiring all 5 would make a single failure stop the cluster.
6. A Kubernetes-style controller must react to every change under /services/, including changes made while it was briefly disconnected. Which feature fits best?
etcd keeps history by revision, so a watch can restart from the last one seen without missing anything. One-shot watches can miss changes between firing and re-registering.
7. A design takes an etcd lock for each of 40,000 payment requests a second. What is the main problem?
Acquiring and releasing is at least two writes, each committed by a majority with fsync: about 80,000 writes/s. Leases do release crashed holders, and etcd persists data. Partition the work or use database constraints instead.
Score 60% or more and the lesson is marked complete.