Pub/Sub and events
Publish-subscribe and event-driven design: topics, fan-out, consumer groups, choreography versus orchestration, and the transactional outbox that keeps events and data in step.
On this page
The problem: the order service that knows too much
When an order is placed, five other parts of the business care: email sends a receipt, the warehouse reserves stock, analytics records the sale, loyalty adds points, fraud checks the payment. The simple approach is for the order service to call each one directly.
Calling every interested service directly couples them all to checkout
- When an order is placed, lots of other services care: email, warehouse, analytics, rewards points. The simple way is for the order service to call each one directly.
- That creates problems. Checkout waits for every call. If analytics is down, checkout may fail. And every new service means changing and redeploying the order service.
This works at first, then hurts in three ways:
- Latency adds up. Checkout waits for five calls in a row (or the slowest of five in parallel).
- Failures spread. If analytics is down, does checkout fail? Every dependency is a new way to break the most important flow in the business.
- Every new consumer means changing the producer. The fraud team can't start listening without the order team editing, testing and redeploying checkout.
The order service has taken on knowledge that isn't its business: who cares about orders, and what they do about it.
The idea: announce the fact, let others subscribe
One event on a topic; every subscriber gets its own copy
- With publish–subscribe, the order service just announces one event, “order placed”, to a topic. It doesn’t know or care who’s listening.
- Every subscriber gets its own copy of the event and reacts in its own time. One event, many receivers: that is called fan-out.
In publish–subscribe (pub/sub), the order service publishes an event (a record of something that happened, in the past tense: OrderPlaced) to a topic (a named channel, here orders). Any number of subscribers register interest in the topic, and each gets its own copy of every event. One event delivered to many receivers is fan-out.
The publisher knows only the topic. It doesn't know who is listening, how many, or whether they are up.
| Message queue | Pub/sub topic | |
|---|---|---|
| Each message goes to | One consumer (workers compete for jobs) | Every subscription gets a copy |
| Message means | "Please do this job" (a command) | "This happened" (an event) |
| Adding a receiver | Shares the existing work | Gets all events, independently |
| Example | Resize this image | Order 7731 was placed |
Most real systems combine the two, as the next section shows. See message queues for acks, retries and dead-letter queues, which apply to subscribers too.
How it works: subscriptions and consumer groups
A subscriber is usually not one process but a service running on many machines. Two copies of the email service must not both send each receipt. So brokers deliver each event once per subscriber group, and within a group, share the events among its members:
- Kafka: a consumer group reads the topic; each partition (a shard of the topic) is assigned to one consumer in the group. The
emailgroup and theanalyticsgroup each see every event; withinemail, 3 consumers split 3 partitions. - Google Cloud Pub/Sub: each subscription gets every message; the many clients pulling from one subscription share them.
- AWS SNS + SQS: SNS topic fans out to one SQS queue per service; that service's workers compete on its queue.
A consumer group shares partitions; the partition count caps its parallelism
So pub/sub between services, competing consumers within a service. Ordering follows the same rule as queues: Kafka keeps order within a partition, so key events by the entity (order_id) to keep each order's events in sequence.
Worked example: sizing the order events
2,000 orders/s at peak, each OrderPlaced event about 1 KB, 5 subscriber groups.
| Quantity | Calculation | Result |
|---|---|---|
| Write rate into the topic | 2,000 × 1 KB | 2 MB/s |
| Read rate out of the brokers | 2 MB/s × 5 groups | 10 MB/s |
| Storage for 7 days of retention | 2 MB/s × 604,800 s ≈ 1.2 TB, × 3 replicas | ≈ 3.6 TB |
| Partitions | A partition handles tens of MB/s, but consumer parallelism is capped by partition count | 12–24 partitions, for headroom and parallel consumers |
The fan-out cost is on reads: every new subscriber group adds another full read of the stream, which brokers handle well because they serve recent data from memory.
Event-driven architecture
Loosely coupled services react to events in their own time
- Now the pieces are loosely coupled. If analytics is down, its events wait for it, and checkout carries on. Want a new fraud checker? Just subscribe it, and the order service never changes.
- Systems built this way are called event-driven. The trade-off: it’s harder to follow what happens after an event, and subscribers see results a moment later, not instantly. Tools like Kafka, Google Pub/Sub and SNS do this at huge scale.
When services communicate mainly by publishing and reacting to events, the system is event-driven. The benefits are those of the figure: a slow or failed subscriber delays only itself (its events wait in the broker), and new features plug in by subscribing.
Events come in a few flavours, and naming which one you mean avoids confusion:
| Kind | Event carries | Subscriber then | Trade-off |
|---|---|---|---|
| Event notification | Just the fact and an id: {"type":"OrderPlaced","order_id":7731} | Calls the order service for details | Small events; but subscribers depend on the producer's API at read time |
| Event-carried state | The data subscribers need: items, totals, address | Acts without calling back | Bigger events, a schema many teams depend on |
| Event sourcing | Every change, stored forever as the source of truth | Rebuilds state by replaying | Powerful audit and replay; complex, a different way to build the whole service |
Choreography vs orchestration
With choreography, each service reacts to events and emits its own (OrderPlaced → warehouse emits StockReserved → payment emits PaymentCaptured…). Nobody is in charge. With orchestration, one coordinator (an order workflow, perhaps using a tool like Temporal or AWS Step Functions) tells each service what to do and tracks the state.
Choreography is great for independent side effects (email, analytics, loyalty). For a multi-step business process with compensations ("if payment fails, release the stock"), orchestration is easier to follow, debug and change. This coordinated sequence of local steps, each with an undo action, is called a saga.
The transactional outbox
The most important correctness problem in event-driven systems: the service must update its database and publish an event. These are two different systems, and there is no transaction across both.
The dual-write problem, and the transactional outbox that fixes it
- Commit, then publish: a crash between the two loses the event. The order exists, but the warehouse never reserves stock.
- Publish, then commit: if the commit fails, subscribers act on an order that doesn't exist.
The transactional outbox pattern fixes this:
BEGIN;
INSERT INTO orders (id, customer_id, total_cents, status)
VALUES (7731, 42, 4000, 'placed');
INSERT INTO outbox (event_id, topic, key, payload)
VALUES ('e5b1…', 'orders', '7731', '{"type":"OrderPlaced","order_id":7731,...}');
COMMIT;
- The order row and the event row are written in one local transaction: both or neither.
- A separate relay publishes outbox rows to the broker. Either it polls the table (
SELECT … WHERE published_at IS NULL ORDER BY id LIMIT 100), or a change data capture (CDC) tool such as Debezium reads the database's change log (Postgres WAL, MySQL binlog) and publishes each new outbox row. - After the broker acknowledges, the relay marks the row as published (or the CDC tool records its position), and old rows are deleted later.
The relay can crash after publishing and before marking, so an event can be published twice. The outbox gives at-least-once, never "exactly once". Consumers must therefore be idempotent: store processed event_ids with a unique constraint in the same transaction as their own changes (sometimes called an inbox table).
Schemas: the contract nobody signs
An event published to a topic is a public API: other teams' code depends on its fields. Renaming a field breaks consumers you may not even know about. Treat event schemas like API versions: use a schema format with compatibility rules (Avro, Protobuf, JSON Schema) and a schema registry that rejects incompatible changes at publish time. Allowed changes are additive (new optional fields with defaults); removing or retyping a field needs a new event type or topic version. Include event_id, event_type, occurred_at, the entity id, and a version in every event.
Options: which broker
| Tool | Model | Good at | Watch out for |
|---|---|---|---|
| Kafka | Partitioned, replicated log with retention; consumer groups track offsets | Very high throughput, ordering per key, replay | Partition count caps group parallelism; heavier to operate |
| Google Cloud Pub/Sub | Managed topics and subscriptions; per-message acks | No operations, autoscaling, push or pull | Ordering only with ordering keys; limited replay (seek to a time or snapshot) |
| AWS SNS + SQS | SNS fans out to SQS queues, one per subscriber | Simple fan-out with durable per-service queues | No replay; FIFO variants needed for ordering |
| Redis Pub/Sub | Fire-and-forget broadcast to connected clients | Very low latency for live notifications | No persistence: a disconnected subscriber misses messages. Redis Streams adds persistence and groups |
When not to use pub/sub
- The caller needs an answer: "is this coupon valid?" is a request, not an event. Use a synchronous call.
- There is exactly one consumer and always will be: a plain queue (or a direct call) is simpler.
- Strong consistency across services is required right now: events make other services eventually consistent. If the warehouse must reserve stock before you confirm the order, that is a synchronous step or an orchestrated workflow.
- Small team, small system: event-driven flows are harder to trace. Without correlation ids and distributed tracing, "why didn't the email go out?" becomes an archaeology project.
In the interview
A marketplace stores listings in PostgreSQL and serves search from Elasticsearch. Sellers edit listings, and today the API writes to Postgres and then to Elasticsearch directly. Sometimes search shows old prices for hours. Redesign it with events.
Compare with our answer
- The direct double write is the dual-write problem: if the Elasticsearch call fails or the process dies, the index stays stale with nothing to repair it.
- Use CDC (Debezium on the Postgres WAL) or an outbox to publish
ListingChangedevents to a topic keyed bylisting_id, so each listing's changes stay in order. - An indexer consumer group writes to Elasticsearch, idempotently (index by listing id with the listing's version; Elasticsearch's external versioning rejects older versions).
- Alert on consumer lag; send unindexable documents to a dead-letter topic.
- To rebuild the index, re-publish a snapshot of all listings (or replay retained events) into a fresh index, then switch an alias.
- Other consumers (cache invalidation, recommendations) can subscribe to the same topic later without touching the API.
A strong answer mentions: why the direct dual write loses updates · outbox or CDC as the source of events · partitioning by listing id · idempotent, version-aware indexing · consumer lag alerts and a dead-letter topic · how to rebuild the index.
A team's order service does: db.commit(order), then broker.publish("OrderPlaced"). Their loyalty service consumes events and runs UPDATE accounts SET points = points + 10 WHERE user_id = ?. Some customers never get points; others get double points. Explain both and fix them.
Compare with our answer
- Missing points: a crash or broker error between the commit and the publish loses the event. Fix with a transactional outbox (or CDC), which guarantees the event is eventually published.
- Double points: delivery is at-least-once (relay retries, consumer crash before committing its offset), and
points = points + 10is not idempotent. Fix: aprocessed_events(event_id PRIMARY KEY)table; in one transaction, insert the event id and update points; a duplicate fails the insert and is skipped. Or record points as rows keyed byorder_idwith a unique constraint, and sum them. - Key the topic by
order_id(oruser_id) so related events stay ordered, and handleOrderCancelledto reverse points.
A strong answer mentions: the dual-write gap between commit and publish · the outbox or CDC fix · at-least-once delivery causing duplicates · a non-idempotent increment · dedupe on event id in the same transaction · ordering and the cancellation event.
- Pub/sub: publisher sends an event (past tense fact) to a topic; every subscriber group gets a copy (fan-out). Queue: each job to one worker.
- Within a subscriber: consumer group (Kafka) / subscription (Google Pub/Sub) / SQS queue behind SNS: members share the events.
- Order only per partition: key by entity id. Partition count caps a group's parallelism.
- Benefits: loose coupling, failures isolated, new consumers without changing the producer. Costs: eventual consistency, harder tracing.
- Event notification vs event-carried state vs event sourcing. Choreography for side effects; orchestration for multi-step processes.
- Dual-write problem → transactional outbox (order + outbox row in one transaction; relay or CDC publishes). At-least-once.
- Consumers: idempotent (dedupe on
event_id), version-aware, dead-letter topic for poison events, alert on consumer lag before retention runs out. - Treat event schemas as APIs: additive changes, schema registry.
1. An order service calls email, warehouse, analytics and loyalty directly. Analytics is down and checkouts start failing. What does pub/sub change?
The publisher only depends on the broker. Each subscriber's events wait for it, so one subscriber's outage doesn't reach checkout.
2. A service commits an order to its database, then publishes OrderPlaced. Occasionally the warehouse never hears about an order. What is the most reliable fix?
Two separate writes can't be atomic. The outbox makes the event part of the database transaction, and the relay publishes it eventually. Publishing first creates phantom events; retries in the handler die with the process.
3. The email service runs 6 instances in one Kafka consumer group on a topic with 4 partitions. What happens?
Within a group, each partition goes to one consumer. With 4 partitions, at most 4 consumers are busy. A different group would get its own copy of every event.
4. A loyalty consumer runs points = points + 10 for each OrderPlaced event. Some users get double points. What is the right fix?
Delivery is at-least-once, so the handler must be idempotent. Deduping on the event id atomically with the update makes repeats harmless. Committing first just loses events instead.
5. OrderCancelled is sometimes processed before OrderPlaced for the same order. What is the most likely cause and fix?
Ordering holds only within a partition. Keying by order id keeps one order's events together and in sequence.
6. A Kafka consumer group has been failing silently for 9 days. Retention is 7 days. What has happened, and what should have prevented it?
Kafka deletes by retention, regardless of who has read. Consumer lag alerts catch a stuck group long before retention runs out.
7. A team must reserve stock, charge the card and arrange shipping, undoing earlier steps if a later one fails. Which approach is easiest to follow and change?
Multi-step processes with compensations are clearer with a coordinator that holds the state. Choreography spreads the logic across services, and a cross-service database transaction isn't available.
Score 60% or more and the lesson is marked complete.