Technology deep dives · Chapter 31

ZooKeeper and etcd

How coordination services work: consensus with ZAB and Raft, leader election, locks with fencing tokens, ephemeral nodes and watches, etcd today, and why Redis locks need fencing.

AdvancedInterview priority: Medium35 min
On this page
  1. The problem: machines that must agree
  2. Consensus, intuitively
  3. ZooKeeper's data model: znodes, sessions and watches
  4. The recipes
  5. Locks need fencing tokens
  6. etcd: the modern alternative
  7. Performance and limits
  8. When to use a coordination service, and when not
  9. In the interview
  10. Quiz

The problem: machines that must agree

A billing service runs on three servers so that one can fail. Exactly one of them must run the nightly invoice job; if two run it, customers are charged twice. A database cluster must agree on which node is the primary. A fleet of workers needs to know which workers are alive and which config version is current.

The obvious solutions fail in the usual ways. A row in a database saying "server B is leader" is a single point of failure, and nothing clears it when server B dies. A heartbeat between the three servers breaks during a network cut: each side thinks the other is dead and both become leader (split brain). Clocks drift, processes pause, and messages arrive late.

Coordination services solve this once, carefully, so every other system can rely on them. Apache ZooKeeper (from Yahoo, around 2008) and etcd (from CoreOS, 2013, now the store behind every Kubernetes cluster) are the two you should know. They are small, strongly consistent, highly available stores for small amounts of critical data: who is leader, who holds a lock, which servers are members, what the config says. They are not for application data.

Consensus, intuitively

Both services run on an odd number of servers (an ensemble or cluster: 3 or 5) and use a consensus protocol so that all servers agree on one ordered log of changes, even if some crash. ZooKeeper uses ZAB (ZooKeeper Atomic Broadcast); etcd uses Raft. They differ in detail but share the same core ideas:

  1. One server is the leader; all writes go through it, which gives a single order.
  2. A write is committed once a majority (a quorum) of servers has stored it on disk. Then it can never be lost, because any future majority overlaps with this one in at least one server.
  3. If the leader dies, the others hold an election. Each new leader has a higher term (Raft) or epoch (ZAB), and a server only votes for a candidate whose log is at least as up to date as its own, so the winner has every committed write.
  4. Messages from a leader with an old term are rejected, which stops a deposed leader doing damage.

Writes commit on a majority; a minority side cannot commit anything

That is why ensembles have odd sizes: 3 servers tolerate 1 failure, 5 tolerate 2, and a 4th or 6th server adds no extra tolerance (4 still needs 3 for a majority). More servers make writes slower, because each needs more acknowledgements, so 3 or 5 is normal and 7 is rare. The same idea underlies split-brain prevention in replication.

ZooKeeper's data model: znodes, sessions and watches

ZooKeeper stores a tree of znodes, like a file system where every node can hold a small amount of data (1 MB at most by default; in practice keep it to a few KB) and children. Each znode has a version, and writes can be conditional on it ("set if version is still 7"), which is an atomic compare-and-set.

Three features make coordination recipes possible:

  • Sessions and ephemeral znodes. A client holds a session, kept alive by heartbeats. An ephemeral znode is deleted automatically when the session that created it ends, whether the client closed it or simply stopped heartbeating for the session timeout (typically a few seconds to tens of seconds). "This node exists" therefore means "its owner is alive".
  • Sequential znodes. ZooKeeper appends an increasing counter to the name: create /election/n_ and get /election/n_0000000007. This gives a fair queue.
  • Watches. A client can ask to be notified when a znode changes or its children change. Classic watches fire once and must be set again; ZooKeeper 3.6 added persistent and recursive watches.

Consistency: writes are linearizable (one total order, as if on one machine). Reads are served by whichever server the client is connected to, so they can be slightly behind the leader, but each client sees changes in order and never goes back in time. Call sync before a read when you need the very latest value.

The recipes

Leader election

Ephemeral sequential nodes: the lowest is leader; each watches only its predecessor

Every candidate creates an ephemeral sequential node; the lowest number leads. Watching only the predecessor (instead of everyone watching the leader) avoids the herd effect, where one change wakes hundreds of clients at once. If a node in the middle dies, only its successor re-checks and starts watching the next one down.

Membership and configuration

  • Group membership: each worker creates an ephemeral node /workers/w-17; anyone can list the children of /workers to see who is alive and watch it for changes. Kafka (before KRaft), HBase and Solr have used ZooKeeper exactly this way.
  • Configuration: store /config/feature-flags and have services watch it. Updates reach all watchers within moments, and the version number tells them which config they have.

Locks

A lock is the election recipe with a different name: create /locks/invoice-run/n_, and you hold the lock when your node is the lowest. Release by deleting it; a crash releases it automatically when the session expires. That expiry is the catch.

Locks need fencing tokens

Every distributed lock is really a lease: it is held for as long as the holder stays alive in the eyes of the lock service. A holder can lose its lease without knowing it: a garbage-collection pause of several seconds, a VM migration, a slow disk, or a network cut long enough for the session to expire. When it resumes, it carries on as if it still held the lock.

A paused client wakes with an old token, and the storage rejects its write

The fix is a fencing token: a number that increases every time the lock is granted, sent with every write to the protected resource, which rejects any token lower than the highest it has seen. Both services provide one: in ZooKeeper, the zxid (transaction ID) or the sequence number of the lock node; in etcd, the revision at which the lock was acquired.

func runInvoices(ctx context.Context, cli *clientv3.Client, store *Store) error {
	sess, err := concurrency.NewSession(cli, concurrency.WithTTL(10))
	if err != nil {
		return err
	}
	defer sess.Close()

	m := concurrency.NewMutex(sess, "/locks/invoice-run")
	if err := m.Lock(ctx); err != nil {
		return err
	}
	defer m.Unlock(ctx)

	token := m.Header().Revision // grows with every acquisition: the fencing token
	return store.RunIfNewer(ctx, "invoice-run", token)
}

The resource must check it, for example in SQL:

UPDATE job_state SET fence = :token, started_at = now()
WHERE job = 'invoice-run' AND fence < :token;      -- 0 rows updated = you lost the lock

etcd: the modern alternative

etcd is a key-value store run on Raft, written in Go, with a gRPC API. The concepts map closely onto ZooKeeper's:

NeedZooKeeperetcd
Data modelTree of znodesFlat keys; "directories" are key prefixes queried as ranges
LivenessSession + ephemeral znodeLease with a TTL, kept alive by the client; keys attached to it vanish when it expires
Change notificationWatches (one-shot; persistent since 3.6)Watch streams on a key or prefix, starting from any past revision, so no change is missed between reconnects
Atomic multi-stepmulti (a batch of operations, all or nothing)Transactions: if (compare) then (ops) else (ops)
VersionsPer-znode version, global zxidEvery change bumps a global revision; MVCC keeps history until compacted
ReadsLocal server (may lag); sync to catch upLinearizable by default (the leader confirms it is still leader); serializable reads optional for speed

etcd's watch-from-revision is a real improvement: with one-shot watches, a change between a notification and setting the next watch could be missed unless you re-read; with etcd you resume from the last revision you saw. That is how Kubernetes controllers keep up with cluster state.

Which one? For a new system, etcd is the usual default, especially near Kubernetes. ZooKeeper remains in many existing systems (HBase, Solr, Hadoop, older Kafka). Kafka itself replaced ZooKeeper with its own Raft-based controllers (KRaft) and removed ZooKeeper support in Kafka 4.0, the same move many systems have made to avoid running a separate ensemble. Consul (also Raft) is a third option, popular for service discovery.

Performance and limits

MeasureTypical value
Write latencya few ms in one region, bounded by disk fsync and a majority round trip
Write throughputthousands to a few tens of thousands of writes/s per cluster; it does not grow with more servers
Readshigher; ZooKeeper reads scale with servers (and observers), etcd linearizable reads go through the leader check
Data sizeZooKeeper: everything in memory, a few GB at most; etcd: 2 GB default quota, ~8 GB recommended maximum, 1.5 MiB per request
Failovera new leader within roughly 1–10 s, depending on timeouts

Worked example. 2,000 workers each renew a lease every 3 seconds (≈ 670 writes/s), plus 50 leader elections and config changes a minute: comfortably within one 3-node etcd cluster. But 50,000 jobs a second each taking and releasing a lock is 100,000 writes/s, beyond any single consensus cluster: partition the work (Kafka partitions or sharding give each worker exclusive keys) instead of locking per job.

When to use a coordination service, and when not

ZooKeeper / etcdRedis lockDatabase row lock or constraintKafka partitions
Good atLeader election, membership, config, correct locks with fencingFast, cheap, best-effort mutual exclusionExclusivity for data in that databaseExclusive ownership of a key range by one consumer
Weak atHigh write volume, large dataCorrectness across failover; no fencingCoordination outside that databaseFine-grained, ad-hoc locks
Use whenA few critical decisions must be agreedDuplicate work is harmlessThe protected data lives in that databaseWork can be divided by key

When not to use one: as a general database or queue (both are small and write-limited); for per-request locks at high volume; when the protected data already lives in a database that can enforce the rule itself (SELECT … FOR UPDATE, a unique index); and when your platform already provides leader election (Kubernetes leases, a managed database's failover), which is usually built on one of these anyway.

In the interview

Your turn: one scheduler, three servers

A cron-like scheduler runs on three servers for availability. Every minute, the active one reads due jobs from PostgreSQL and enqueues them. A job must never be enqueued twice. Design the leader election, what happens when the leader dies or pauses, and how duplicate enqueues are prevented.

Compare with our answer

Run a 3-node etcd (or use the platform's lease API). Each scheduler campaigns for /scheduler/leader with a lease of ~10 s, renewed every ~3 s; the winner gets the election revision as its fencing token. When the leader dies, its lease expires within ~10 s and another instance wins; schedules a minute apart tolerate that gap. To protect against a paused leader, enqueueing is guarded twice: the scheduler claims each run with UPDATE runs SET claimed_by_token = :token WHERE job_id = ? AND due_at = ? AND claimed_by_token < :token, or inserts into a table with a unique constraint on (job_id, due_at), so a second enqueue of the same run fails whoever does it. Workers also treat jobs idempotently.

A strong answer mentions: election with a lease and its TTL · the fencing token from the coordination service · the resource rejecting old tokens · a unique constraint or idempotency as a second guard · the failover gap and why it is acceptable · etcd's 3-node majority.

Your turn: spot the bugs

A team stores per-user shopping carts in ZooKeeper "because it is strongly consistent", one znode per user, 10 million users. Each service instance sets a watch on /config and on the leader's znode; on every change, all 800 instances re-read everything. Their lock is a single znode /lock: whoever creates it holds it, everyone else watches it. The ensemble has 4 servers "for extra safety". What would you change?

Compare with our answer

A strong answer mentions:

  • Carts are application data: 10 M znodes in memory and constant writes overload a store meant for small coordination data; carts belong in a database or Redis.
  • A single /lock node that everyone watches causes a herd effect: each release wakes all waiters, who all race to create it; use ephemeral sequential nodes with each waiter watching its predecessor.
  • The lock has no fencing token, so a paused holder can still write after its session expires; pass the zxid or sequence number and check it at the resource.
  • 4 servers tolerate only 1 failure (like 3) but add write latency; use 3 or 5.
  • 800 instances re-reading everything on each change is a burst of reads on every update; read only the changed node, or use etcd watches from a revision.
ZooKeeper and etcd on one page
  • Small, strongly consistent, highly available stores for coordination data: leaders, locks, membership, config. Not a database.
  • Consensus (ZAB in ZooKeeper, Raft in etcd): one leader orders writes; commit on a majority; elections with increasing terms/epochs.
  • 3 nodes tolerate 1 failure, 5 tolerate 2; even sizes add nothing; more nodes = slower writes.
  • ZooKeeper: znode tree, versions for compare-and-set, ephemeral nodes tied to sessions, sequential nodes, watches (one-shot; persistent since 3.6). Reads may lag; sync to catch up.
  • etcd: flat keys, leases, watch streams from a revision, if/then/else transactions, linearizable reads by default. Kubernetes runs on it.
  • Election and locks: lowest ephemeral sequential node wins; watch only the predecessor (no herd).
  • Every lock is a lease: a paused holder can lose it unknowingly. Fencing tokens (zxid, etcd revision), checked by the resource, make it safe.
  • Redis locks: fine for efficiency; for correctness use consensus + fencing, or a database constraint.
  • Throughput: thousands to tens of thousands of writes/s; keep it off the per-request path.
Check what stuck

1. A team runs a 4-node ZooKeeper ensemble instead of 3 "to survive more failures". How many servers can fail while writes still succeed?

2. A worker holding a ZooKeeper lock pauses for 30 seconds in garbage collection. Its session times out, another worker takes the lock, and then the first worker resumes. What prevents both writing to the database?

3. 500 clients wait for one lock, and every client watches the single lock znode. What happens each time the lock is released?

4. Why is a lock built with SET key token NX PX 10000 on a single Redis primary with one replica unsafe for correctness?

5. A network cut splits a 5-node etcd cluster into groups of 2 (including the old leader) and 3. What happens?

6. A Kubernetes-style controller must react to every change under /services/, including changes made while it was briefly disconnected. Which feature fits best?

7. A design takes an etcd lock for each of 40,000 payment requests a second. What is the main problem?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.