Talking between services · Chapter 12

Pub/Sub and events

Publish-subscribe and event-driven design: topics, fan-out, consumer groups, choreography versus orchestration, and the transactional outbox that keeps events and data in step.

AdvancedInterview priority: High35 min
On this page
  1. The problem: the order service that knows too much
  2. The idea: announce the fact, let others subscribe
  3. How it works: subscriptions and consumer groups
  4. Event-driven architecture
  5. The transactional outbox
  6. Options: which broker
  7. When not to use pub/sub
  8. In the interview
  9. Quiz

The problem: the order service that knows too much

When an order is placed, five other parts of the business care: email sends a receipt, the warehouse reserves stock, analytics records the sale, loyalty adds points, fraud checks the payment. The simple approach is for the order service to call each one directly.

Calling every interested service directly couples them all to checkout

  1. When an order is placed, lots of other services care: email, warehouse, analytics, rewards points. The simple way is for the order service to call each one directly.
  2. That creates problems. Checkout waits for every call. If analytics is down, checkout may fail. And every new service means changing and redeploying the order service.

This works at first, then hurts in three ways:

  • Latency adds up. Checkout waits for five calls in a row (or the slowest of five in parallel).
  • Failures spread. If analytics is down, does checkout fail? Every dependency is a new way to break the most important flow in the business.
  • Every new consumer means changing the producer. The fraud team can't start listening without the order team editing, testing and redeploying checkout.

The order service has taken on knowledge that isn't its business: who cares about orders, and what they do about it.

The idea: announce the fact, let others subscribe

One event on a topic; every subscriber gets its own copy

  1. With publish–subscribe, the order service just announces one event, “order placed”, to a topic. It doesn’t know or care who’s listening.
  2. Every subscriber gets its own copy of the event and reacts in its own time. One event, many receivers: that is called fan-out.

In publish–subscribe (pub/sub), the order service publishes an event (a record of something that happened, in the past tense: OrderPlaced) to a topic (a named channel, here orders). Any number of subscribers register interest in the topic, and each gets its own copy of every event. One event delivered to many receivers is fan-out.

The publisher knows only the topic. It doesn't know who is listening, how many, or whether they are up.

Message queuePub/sub topic
Each message goes toOne consumer (workers compete for jobs)Every subscription gets a copy
Message means"Please do this job" (a command)"This happened" (an event)
Adding a receiverShares the existing workGets all events, independently
ExampleResize this imageOrder 7731 was placed

Most real systems combine the two, as the next section shows. See message queues for acks, retries and dead-letter queues, which apply to subscribers too.

How it works: subscriptions and consumer groups

A subscriber is usually not one process but a service running on many machines. Two copies of the email service must not both send each receipt. So brokers deliver each event once per subscriber group, and within a group, share the events among its members:

  • Kafka: a consumer group reads the topic; each partition (a shard of the topic) is assigned to one consumer in the group. The email group and the analytics group each see every event; within email, 3 consumers split 3 partitions.
  • Google Cloud Pub/Sub: each subscription gets every message; the many clients pulling from one subscription share them.
  • AWS SNS + SQS: SNS topic fans out to one SQS queue per service; that service's workers compete on its queue.

A consumer group shares partitions; the partition count caps its parallelism

So pub/sub between services, competing consumers within a service. Ordering follows the same rule as queues: Kafka keeps order within a partition, so key events by the entity (order_id) to keep each order's events in sequence.

Worked example: sizing the order events

2,000 orders/s at peak, each OrderPlaced event about 1 KB, 5 subscriber groups.

QuantityCalculationResult
Write rate into the topic2,000 × 1 KB2 MB/s
Read rate out of the brokers2 MB/s × 5 groups10 MB/s
Storage for 7 days of retention2 MB/s × 604,800 s ≈ 1.2 TB, × 3 replicas≈ 3.6 TB
PartitionsA partition handles tens of MB/s, but consumer parallelism is capped by partition count12–24 partitions, for headroom and parallel consumers

The fan-out cost is on reads: every new subscriber group adds another full read of the stream, which brokers handle well because they serve recent data from memory.

Event-driven architecture

Loosely coupled services react to events in their own time

  1. Now the pieces are loosely coupled. If analytics is down, its events wait for it, and checkout carries on. Want a new fraud checker? Just subscribe it, and the order service never changes.
  2. Systems built this way are called event-driven. The trade-off: it’s harder to follow what happens after an event, and subscribers see results a moment later, not instantly. Tools like Kafka, Google Pub/Sub and SNS do this at huge scale.

When services communicate mainly by publishing and reacting to events, the system is event-driven. The benefits are those of the figure: a slow or failed subscriber delays only itself (its events wait in the broker), and new features plug in by subscribing.

Events come in a few flavours, and naming which one you mean avoids confusion:

KindEvent carriesSubscriber thenTrade-off
Event notificationJust the fact and an id: {"type":"OrderPlaced","order_id":7731}Calls the order service for detailsSmall events; but subscribers depend on the producer's API at read time
Event-carried stateThe data subscribers need: items, totals, addressActs without calling backBigger events, a schema many teams depend on
Event sourcingEvery change, stored forever as the source of truthRebuilds state by replayingPowerful audit and replay; complex, a different way to build the whole service

Choreography vs orchestration

With choreography, each service reacts to events and emits its own (OrderPlaced → warehouse emits StockReserved → payment emits PaymentCaptured…). Nobody is in charge. With orchestration, one coordinator (an order workflow, perhaps using a tool like Temporal or AWS Step Functions) tells each service what to do and tracks the state.

Choreography is great for independent side effects (email, analytics, loyalty). For a multi-step business process with compensations ("if payment fails, release the stock"), orchestration is easier to follow, debug and change. This coordinated sequence of local steps, each with an undo action, is called a saga.

The transactional outbox

The most important correctness problem in event-driven systems: the service must update its database and publish an event. These are two different systems, and there is no transaction across both.

The dual-write problem, and the transactional outbox that fixes it

  • Commit, then publish: a crash between the two loses the event. The order exists, but the warehouse never reserves stock.
  • Publish, then commit: if the commit fails, subscribers act on an order that doesn't exist.

The transactional outbox pattern fixes this:

BEGIN;
INSERT INTO orders (id, customer_id, total_cents, status)
VALUES (7731, 42, 4000, 'placed');
INSERT INTO outbox (event_id, topic, key, payload)
VALUES ('e5b1…', 'orders', '7731', '{"type":"OrderPlaced","order_id":7731,...}');
COMMIT;
  1. The order row and the event row are written in one local transaction: both or neither.
  2. A separate relay publishes outbox rows to the broker. Either it polls the table (SELECT … WHERE published_at IS NULL ORDER BY id LIMIT 100), or a change data capture (CDC) tool such as Debezium reads the database's change log (Postgres WAL, MySQL binlog) and publishes each new outbox row.
  3. After the broker acknowledges, the relay marks the row as published (or the CDC tool records its position), and old rows are deleted later.

The relay can crash after publishing and before marking, so an event can be published twice. The outbox gives at-least-once, never "exactly once". Consumers must therefore be idempotent: store processed event_ids with a unique constraint in the same transaction as their own changes (sometimes called an inbox table).

Schemas: the contract nobody signs

An event published to a topic is a public API: other teams' code depends on its fields. Renaming a field breaks consumers you may not even know about. Treat event schemas like API versions: use a schema format with compatibility rules (Avro, Protobuf, JSON Schema) and a schema registry that rejects incompatible changes at publish time. Allowed changes are additive (new optional fields with defaults); removing or retyping a field needs a new event type or topic version. Include event_id, event_type, occurred_at, the entity id, and a version in every event.

Options: which broker

ToolModelGood atWatch out for
KafkaPartitioned, replicated log with retention; consumer groups track offsetsVery high throughput, ordering per key, replayPartition count caps group parallelism; heavier to operate
Google Cloud Pub/SubManaged topics and subscriptions; per-message acksNo operations, autoscaling, push or pullOrdering only with ordering keys; limited replay (seek to a time or snapshot)
AWS SNS + SQSSNS fans out to SQS queues, one per subscriberSimple fan-out with durable per-service queuesNo replay; FIFO variants needed for ordering
Redis Pub/SubFire-and-forget broadcast to connected clientsVery low latency for live notificationsNo persistence: a disconnected subscriber misses messages. Redis Streams adds persistence and groups

When not to use pub/sub

  • The caller needs an answer: "is this coupon valid?" is a request, not an event. Use a synchronous call.
  • There is exactly one consumer and always will be: a plain queue (or a direct call) is simpler.
  • Strong consistency across services is required right now: events make other services eventually consistent. If the warehouse must reserve stock before you confirm the order, that is a synchronous step or an orchestrated workflow.
  • Small team, small system: event-driven flows are harder to trace. Without correlation ids and distributed tracing, "why didn't the email go out?" becomes an archaeology project.

In the interview

Your turn: keep the search index in sync

A marketplace stores listings in PostgreSQL and serves search from Elasticsearch. Sellers edit listings, and today the API writes to Postgres and then to Elasticsearch directly. Sometimes search shows old prices for hours. Redesign it with events.

Compare with our answer
  • The direct double write is the dual-write problem: if the Elasticsearch call fails or the process dies, the index stays stale with nothing to repair it.
  • Use CDC (Debezium on the Postgres WAL) or an outbox to publish ListingChanged events to a topic keyed by listing_id, so each listing's changes stay in order.
  • An indexer consumer group writes to Elasticsearch, idempotently (index by listing id with the listing's version; Elasticsearch's external versioning rejects older versions).
  • Alert on consumer lag; send unindexable documents to a dead-letter topic.
  • To rebuild the index, re-publish a snapshot of all listings (or replay retained events) into a fresh index, then switch an alias.
  • Other consumers (cache invalidation, recommendations) can subscribe to the same topic later without touching the API.

A strong answer mentions: why the direct dual write loses updates · outbox or CDC as the source of events · partitioning by listing id · idempotent, version-aware indexing · consumer lag alerts and a dead-letter topic · how to rebuild the index.

Your turn: spot the bug

A team's order service does: db.commit(order), then broker.publish("OrderPlaced"). Their loyalty service consumes events and runs UPDATE accounts SET points = points + 10 WHERE user_id = ?. Some customers never get points; others get double points. Explain both and fix them.

Compare with our answer
  • Missing points: a crash or broker error between the commit and the publish loses the event. Fix with a transactional outbox (or CDC), which guarantees the event is eventually published.
  • Double points: delivery is at-least-once (relay retries, consumer crash before committing its offset), and points = points + 10 is not idempotent. Fix: a processed_events(event_id PRIMARY KEY) table; in one transaction, insert the event id and update points; a duplicate fails the insert and is skipped. Or record points as rows keyed by order_id with a unique constraint, and sum them.
  • Key the topic by order_id (or user_id) so related events stay ordered, and handle OrderCancelled to reverse points.

A strong answer mentions: the dual-write gap between commit and publish · the outbox or CDC fix · at-least-once delivery causing duplicates · a non-idempotent increment · dedupe on event id in the same transaction · ordering and the cancellation event.

Pub/sub and events on one page
  • Pub/sub: publisher sends an event (past tense fact) to a topic; every subscriber group gets a copy (fan-out). Queue: each job to one worker.
  • Within a subscriber: consumer group (Kafka) / subscription (Google Pub/Sub) / SQS queue behind SNS: members share the events.
  • Order only per partition: key by entity id. Partition count caps a group's parallelism.
  • Benefits: loose coupling, failures isolated, new consumers without changing the producer. Costs: eventual consistency, harder tracing.
  • Event notification vs event-carried state vs event sourcing. Choreography for side effects; orchestration for multi-step processes.
  • Dual-write problem → transactional outbox (order + outbox row in one transaction; relay or CDC publishes). At-least-once.
  • Consumers: idempotent (dedupe on event_id), version-aware, dead-letter topic for poison events, alert on consumer lag before retention runs out.
  • Treat event schemas as APIs: additive changes, schema registry.
Interactive labPractise it: the pub/sub playgroundOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. An order service calls email, warehouse, analytics and loyalty directly. Analytics is down and checkouts start failing. What does pub/sub change?

2. A service commits an order to its database, then publishes OrderPlaced. Occasionally the warehouse never hears about an order. What is the most reliable fix?

3. The email service runs 6 instances in one Kafka consumer group on a topic with 4 partitions. What happens?

4. A loyalty consumer runs points = points + 10 for each OrderPlaced event. Some users get double points. What is the right fix?

5. OrderCancelled is sometimes processed before OrderPlaced for the same order. What is the most likely cause and fix?

6. A Kafka consumer group has been failing silently for 9 days. Retention is 7 days. What has happened, and what should have prevented it?

7. A team must reserve stock, charge the card and arrange shipping, undoing earlier steps if a later one fails. Which approach is easiest to follow and change?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.