Technology deep dives · Chapter 26

Kafka

How Apache Kafka works: the partitioned log, offsets, consumer groups, replication and ISR, acks, ordering, retention and compaction, exactly-once and KRaft.

AdvancedInterview priority: High40 min
On this page
  1. The problem: many systems need the same stream of events
  2. The log, topics and offsets
  3. Partitions, keys and consumer groups
  4. Replication, ISR and acks
  5. Delivery semantics and exactly-once
  6. Retention and compaction
  7. Choosing the number of partitions
  8. The control plane: KRaft
  9. Performance and comparisons
  10. In the interview
  11. Quiz

The problem: many systems need the same stream of events

An online shop places 5,000 orders a second at peak. Billing must charge each one, the warehouse must pick it, search must update stock counts, analytics wants every event, and a fraud model wants the last hour of orders replayed whenever it is retrained.

With a classic message queue, a message is deleted once one consumer acknowledges it, so five teams need five queues and the order service must write to all of them. A consumer that had a bug last Tuesday cannot go back and reprocess Tuesday. A new team that arrives next month starts with nothing.

Apache Kafka solves this by storing events in a durable, replicated log that consumers read at their own pace. Writing an event once makes it available to any number of consumers, for as long as you keep it (days, months or forever), and each consumer can rewind. A single partition handles tens of MB/s; a cluster handles millions of events a second.

The log, topics and offsets

A topic is a named stream of records, such as orders. Each record has an optional key, a value (bytes: JSON, Avro, Protobuf), a timestamp and optional headers. A topic is split into partitions, and each partition is an append-only log.

A partition is an append-only log; each consumer tracks its own offset

Reading does not delete anything. Each consumer stores its offset (its position: "the next record I will read is offset 3") and moves it forward as it processes. Kafka keeps the committed offsets of each consumer group in an internal topic, __consumer_offsets. Rewinding is just setting the offset back.

On disk, each partition is a series of segment files (1 GB by default). Writes are sequential appends and reads are mostly sequential too, served largely from the operating system's page cache and sent to the network with zero-copy transfers. That is why Kafka is fast on ordinary disks.

Partitions, keys and consumer groups

The producer hashes the key to choose a partition

The producer chooses a partition for each record. With a key, it hashes the key (murmur2) modulo the partition count, so all records with the same key go to the same partition. Without a key, the default partitioner fills a batch for one partition, then moves on ("sticky" partitioning), which spreads load while keeping batches large.

A consumer group is a set of consumers that share the work of reading a topic. Each partition is assigned to exactly one consumer in the group; different groups read independently, each with its own offsets.

Each partition goes to one consumer in a group; extra consumers sit idle

When consumers join, leave or crash, the group rebalances: partitions are reassigned. In older Kafka versions a rebalance paused the whole group; cooperative rebalancing, and the new consumer group protocol (KIP-848, generally available in Kafka 4.0) in which the broker drives assignment, move only the partitions that need to move.

Ordering: per partition only

Kafka guarantees order within a partition, not across a topic. If all events for order 981 share the key order-981, billing sees "created, paid, shipped" in that order. Events for different orders, on different partitions, can be processed in any relative order.

Two things can still break per-key order: retries without idempotence (fixed by the idempotent producer, below, which keeps order with up to 5 requests in flight) and changing the partition count, which changes hash(key) mod N and sends a key's new events to a different partition from its old ones.

Replication, ISR and acks

Each partition is copied to several brokers: replication factor 3 is typical. One replica is the leader and handles all writes (and by default reads); the others are followers that fetch from it.

Leader, followers and the in-sync replica set; what acks=all protects

The in-sync replica set (ISR) is the leader plus the followers that are caught up: a follower that has not caught up within replica.lag.time.max.ms (30 s by default) is removed until it catches up again. The producer's acks setting decides when a write counts as done:

acksLeader replies whenCan lose acknowledged writes?Latency
0Never waitsYes, freelyLowest
1The leader has written itYes, if the leader dies before followers copy itLow
all (default since Kafka 3.0)Every replica in the ISR has itNo, provided min.insync.replicas ≥ 2 and unclean election is offA few ms more

acks=all alone is not enough: if the ISR shrinks to just the leader, "all" means one copy. Set min.insync.replicas=2 with RF 3, so writes are refused (NotEnoughReplicas) rather than accepted with a single copy. Keep unclean.leader.election.enable=false (the default), so an out-of-sync replica is never made leader. That combination tolerates one broker failure without losing or refusing writes.

Kafka acknowledges once replicas have the record in memory (page cache), not after an fsync. Durability comes from having several copies on separate machines, so spread replicas across racks or availability zones.

The high watermark: what consumers may read

The leader's log has two important positions. The log end offset (LEO) is where the next record will be appended. The high watermark (HW) is the offset up to which every replica in the ISR has copied the log; records below it are called committed. Followers report how far they have fetched, the leader advances the HW, and it passes the HW back to followers in fetch responses.

Consumers can only read below the high watermark, even though the leader already has the newer records. The reason is failover. Suppose the leader has offsets 0 to 104 but the followers have only reached 100, so the HW is 100. If a consumer were allowed to read offset 103 and the leader then died, a follower that never received 101 to 104 would become leader, and new records would be written at those offsets. The consumer would have acted on a record that, as far as the cluster is concerned, never existed, and offset 103 would now mean something else. Reading only committed records means anything a consumer sees survives the loss of any replica that was in the ISR.

Two consequences worth knowing:

  • A slow follower holds the HW back, which adds end-to-end latency until it is dropped from the ISR (after replica.lag.time.max.ms). This is why a sick broker can make consumers look slow.
  • With acks=1, the producer is acknowledged before the HW moves. If the leader then fails, the record is truncated away: the write was acknowledged but lost, though no consumer ever saw it. With transactions and read_committed, consumers stop even earlier, at the last stable offset, the first offset of a transaction still open.

Delivery semantics and exactly-once

Whether each event is processed once depends on when the consumer commits its offset:

  • At most once: commit, then process. A crash after the commit skips the event.
  • At least once: process, then commit. A crash before the commit processes it again. This is the usual choice, together with idempotent processing.
  • Exactly once: possible, with limits.

Kafka offers two building blocks:

  1. Idempotent producer (on by default since 3.0): each producer has an ID and numbers its records per partition, so a retried send is stored once, not twice.
  2. Transactions: a producer with a transactional.id writes to several partitions and commits the consumer's offsets atomically. Consumers with isolation.level=read_committed see only committed transactions.
// consumer: enable.auto.commit=false, isolation.level=read_committed
// producer: transactional.id set (one per input partition set / instance)
producer.initTransactions();
while (true) {
    ConsumerRecords<String, String> recs = consumer.poll(Duration.ofMillis(200));
    if (recs.isEmpty()) continue;
    try {
        producer.beginTransaction();
        for (ConsumerRecord<String, String> r : recs)
            producer.send(new ProducerRecord<>("invoices", r.key(), toInvoice(r.value())));
        // nextOffsets(recs): for each partition, the offset of its last record + 1
        producer.sendOffsetsToTransaction(nextOffsets(recs), consumer.groupMetadata());
        producer.commitTransaction();          // outputs and offsets commit together
    } catch (ProducerFencedException | OutOfOrderSequenceException e) {
        producer.close();                      // a newer instance took over this transactional.id: stop
        throw e;
    } catch (KafkaException e) {
        producer.abortTransaction();           // nothing from this batch becomes visible
        rewindToLastCommitted(consumer);       // seek back so the batch is processed again
    }
}

Poison pills, retry topics and a dead-letter topic

A poison pill is a record the consumer can never process: bytes that don't deserialise, a field that breaks a validation rule, a reference to a customer that was deleted. Because a consumer commits offsets in order, a naive "retry until it works" loop sticks on that record for ever, and every record behind it on the partition waits. Lag climbs while the consumer looks busy.

The usual fix separates failures by kind:

FailureExampleWhat to do
PermanentCorrupt bytes, schema mismatch, invalid dataSend straight to a dead-letter topic (DLT), commit, move on
TransientDatabase timeout, a downstream service returning 503Retry a few times in place with backoff, then move to a retry topic

A common chain is orders → orders-retry-1m → orders-retry-10m → orders-dlt. Each retry topic's consumer waits until the record's timestamp plus the delay before processing it (pausing its partition, not sleeping per record), and sends it one step further on failure. Copy the record unchanged and add headers: original topic, partition, offset, error and attempt count. The DLT has no automatic consumer: it needs an alert, a dashboard and a tool to replay records once the bug is fixed. Spring Kafka (@RetryableTopic) and Kafka Connect (errors.deadletterqueue.topic.name) build this for you.

Retention and compaction

Each topic has a cleanup policy:

  • Delete (default): segments older than retention.ms (7 days by default) or beyond retention.bytes are deleted. Good for event streams.
  • Compact: Kafka keeps at least the latest record for each key and removes older ones in the background. A record with a null value is a tombstone: it deletes the key, and is itself removed after delete.retention.ms (1 day by default). A compacted topic is a changelog you can replay to rebuild current state, such as "latest address per customer".

Tiered storage (production-ready since Kafka 3.9) moves old segments to object storage while brokers keep recent data on local disk, making long retention much cheaper.

Choosing the number of partitions

Partitions are the unit of parallelism: within one group, at most one consumer per partition.

Worked example. Orders peak at 5,000 records/s of 1 KB = 5 MB/s. Producers reach well over 10 MB/s per partition, but each consumer does a database write per order and manages 400 records/s.

  • Consumer bound: 5,000 / 400 = 12.5, so at least 13 partitions.
  • Growth: plan for 3× in two years: ~38. Pick 48 (a number with many divisors spreads evenly over 6, 8 or 12 consumers).
  • Storage: 5 MB/s × 86,400 s ≈ 430 GB a day at peak rate; × RF 3 × 7 days ≈ 9 TB in the cluster, an upper bound since traffic is not at peak all day.

You can add partitions later but never remove them, and adding them moves keys to new partitions, so over-provision moderately at the start. Too many partitions also costs: more files, more replication traffic, slower leader elections and more memory in clients. Thousands of partitions per broker is normal; tens of thousands strains it.

Autoscaling consumers on lag

Consumer lag is, per partition, the latest readable offset minus the group's committed offset: how many records are waiting. Summed across partitions, it is the best signal for scaling consumers, because CPU can look fine while a consumer waits on a slow database. Tools such as KEDA's Kafka scaler on Kubernetes read the lag and set the number of consumer instances.

With the 48-partition topic above: normal traffic of 5,000 records/s needs 13 consumers at 400 records/s each. A sale pushes traffic to 12,000 records/s and lag reaches 1.8 million records. The autoscaler can go to 48 consumers and no further: a 49th would get no partition and sit idle (KEDA caps at the partition count by default). Capacity is then 48 × 400 = 19,200 records/s, so the backlog drains at 19,200 − 12,000 = 7,200 records/s: 1.8 M / 7,200 = 250 s, about four minutes.

Things that make this work in practice:

  • Scale on time, not just records. 10,000 records of lag is nothing at 20,000/s and an hour at 3/s. Divide lag by the consumption rate to get "seconds behind", and alert on that.
  • Every scale step triggers a rebalance. Cooperative rebalancing and the KIP-848 protocol make it cheap, but use a stabilisation window (minutes, not seconds) so the group doesn't flap.
  • Watch the shared bottleneck. Forty-eight consumers can overwhelm the database they write to; the partition count caps parallelism, and so does whatever sits downstream.
  • The ceiling is set at topic creation, which is another reason to over-provision partitions moderately.

The control plane: KRaft

Kafka used to keep cluster metadata (which brokers exist, which replica leads each partition) in a separate ZooKeeper ensemble. KRaft replaces it with a small quorum of Kafka controllers that store the metadata in an internal Raft-replicated log. KRaft has been production-ready since 3.3, and Kafka 4.0 (2025) removed ZooKeeper entirely. It means one system to run, faster failover of partition leadership, and support for far more partitions per cluster.

Performance and comparisons

MeasureTypical value
Throughput per partitiontens of MB/s
Throughput per brokerhundreds of MB/s, often bounded by network or disk
End-to-end latencya few ms to tens of ms; more when producers wait to fill batches (linger.ms)
Retentionhours to forever; limited by disk or tiered storage cost
KafkaRabbitMQ / SQSRedis StreamsPulsar / Kinesis
Good atHigh-volume durable streams, replay, many consumer groupsPer-message work queues, routing, delays, per-message retriesSmall, fast streams next to an existing RedisSimilar log model; Kinesis fully managed, Pulsar separates storage
Weak atPer-message acks and retries, delayed messages, millions of tiny topicsReplay, long retention, very high fan-outDurability on failover; data bigger than RAMSmaller ecosystems than Kafka
Use whenSeveral teams need the same events, or you need historyA job must be done once by some workerVolume is modest and loss is tolerableYou want managed (Kinesis) or geo features (Pulsar)

When not to use Kafka: a simple background job queue with retries and delays (use SQS or RabbitMQ; Kafka's "share groups", queue-style consumption in Kafka 4.x, narrow this gap but are new); request/response lookups ("what is order 981's status?" belongs in a database); a small system where running a cluster costs more than it saves; and work that needs one slow message to be retried without blocking the partition behind it.

In the interview

Your turn: ride events

A ride-hailing app sends driver location updates every 4 seconds from 300,000 active drivers. Matching needs the latest location per driver; analytics wants all updates for a day; a trip's events (requested, accepted, started, finished) must be processed in order. Design the topics, keys, partition counts and retention.

Compare with our answer

Two topics. driver-locations: 300 K / 4 s = 75,000 records/s of ~200 bytes ≈ 15 MB/s, keyed by driver_id; 30–60 partitions, delete policy with a short retention (a few hours to a day) since stale positions are useless; analytics reads the same topic as its own consumer group, or it is copied to object storage. Optionally a compacted driver-latest topic for rebuilding the latest position per driver. trip-events: keyed by trip_id so each trip's events stay in order; much lower volume; longer retention (7–30 days) for replay. Producers use acks=all for trip events; location updates could use acks=1, since losing one update is harmless.

A strong answer mentions: throughput worked out in records/s and MB/s · the key chosen for ordering · partitions from consumer parallelism · different retention per topic · compaction for "latest per key" · durability settings matched to the value of the data.

Your turn: spot the bugs

A payments team consumes payments with auto-commit enabled every 5 seconds, calls the card processor for each record, and sometimes customers are charged twice or not at all. The topic has RF 2 with acks=1. To scale, they raised partitions from 12 to 24 last week, and now refund events are sometimes processed before the payment they refund. What is wrong?

Compare with our answer

A strong answer mentions:

  • Auto-commit commits offsets on a timer, unrelated to processing: a crash can skip records (not charged) or repeat them (charged twice). Commit after processing, and make the charge idempotent with a payment ID as the idempotency key.
  • Kafka transactions would not help here, because the card charge is outside Kafka.
  • RF 2 with acks=1 can lose acknowledged payment events when a leader fails; use RF 3, acks=all, min.insync.replicas=2.
  • Raising the partition count changed hash(key) mod N, so a payment and its later refund can land on different partitions; plan partitions up front, or drain and migrate keys deliberately.
  • Make sure payment and refund share a key (the payment ID), or ordering is never guaranteed.
Kafka on one page
  • A distributed, replicated, append-only log. Topic → partitions → records with offsets. Reading does not delete.
  • Key → hash(key) mod partitions: same key, same partition, ordered. No order across partitions.
  • Consumer group: each partition read by one consumer; partitions = max parallelism. Offsets stored in __consumer_offsets.
  • RF 3, acks=all, min.insync.replicas=2, no unclean election: survives one broker loss without losing acked writes.
  • ISR: followers within replica.lag.time.max.ms (30 s). Leader elected from the ISR.
  • At-least-once (process, then commit) + idempotent consumers is the norm. Exactly-once = idempotent producer + transactions, only Kafka to Kafka.
  • Retention by time/size (7 days default) or compaction (latest per key, tombstones). Tiered storage for long retention.
  • Size partitions from throughput per consumer, with headroom; you can add but not remove them, and adding moves keys.
  • KRaft replaced ZooKeeper; Kafka 4.0 has no ZooKeeper.
  • Consumers read only below the high watermark (replicated to the whole ISR), so nothing they see vanishes on failover.
  • Poison pills: permanent errors to a DLT, transient ones through delayed retry topics; this breaks per-key order unless you park the key. Autoscale on lag (in seconds), capped at the partition count.
Check what stuck

1. A topic has 6 partitions. The team runs 10 consumers in one consumer group to process it faster. What happens?

2. Events for one bank account must be processed in order. How should the producer publish them?

3. A cluster uses RF 3 and acks=all, with min.insync.replicas=1. Two followers fall behind and leave the ISR, then the leader's disk fails. What can happen?

4. A consumer reads orders from Kafka and inserts them into PostgreSQL. The team enabled Kafka transactions for exactly-once. After a crash, some orders are inserted twice. Why?

5. A topic stores each customer's current address, keyed by customer ID. A new service must rebuild all current addresses at startup. Which configuration fits?

6. Consumers process 500 records/s each. Peak traffic is 9,000 records/s and is expected to double. How many partitions are sensible for the topic?

7. A team doubles a topic's partitions from 24 to 48 while the system is running. What should they expect?

8. One record on a partition fails to deserialise. The consumer retries it in a loop, and lag on that partition grows for hours. What is the best fix?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.