Interview patterns · Design 9

Transactions across services

How to keep data consistent when one action spans several services: why two-phase commit is avoided, sagas with compensations, the transactional outbox and change data capture.

AdvancedInterview priority: High40 min
On this page
  1. The problem: three services, one checkout
  2. Signals that you are facing it
  3. Two-phase commit, and why it's avoided
  4. The saga: local steps plus compensations
  5. The dual-write problem and the transactional outbox
  6. Worked example: a checkout saga failing at step 3
  7. When not to use these tools
  8. Which case studies use it
  9. In the interview
  10. Quiz

The problem: three services, one checkout

A shop has split its monolith into services, each with its own database: Orders, Payments and Inventory. Placing an order must create the order, charge the card and reserve the stock. In the monolith, that was one database transaction: either everything happened or nothing did.

Now there are three databases, and no single COMMIT covers them. Each step can fail independently: the card is declined after the order exists; the stock runs out after the card is charged; the Inventory service times out and you don't know whether it reserved anything. Without care you get customers charged for orders that don't exist, or orders shipped that were never paid for.

This lesson is about the tools for that situation: two-phase commit (and why it's usually avoided), sagas, the transactional outbox, and change data capture.

Signals that you are facing it

  • One user action changes data owned by two or more services: order + payment + stock, booking + payment, ride + driver status.
  • A database write plus a message: "save the order and publish OrderCreated".
  • Calls to external systems that can't join your transaction: payment providers, email, shipping partners.
  • Requirements like "never charge without an order" or "release the seat if payment fails".

Say early: "This spans services, so there's no single transaction. I'll use a saga with compensating steps, and an outbox so each service's database change and its event can't diverge."

Two-phase commit, and why it's avoided

Two-phase commit (2PC) is the classic way to make several databases commit atomically. A coordinator asks every participant to prepare (write the change durably, lock the rows, promise it can commit), and only if all vote yes does it tell them to commit.

Two-phase commit, and the coordinator that blocks everyone

It works, and it is used: inside distributed databases such as Spanner and CockroachDB, where each participant and the coordinator's state are themselves replicated with consensus, so a single crashed machine doesn't block anyone for long. Across independent services, it is avoided because:

  • Blocking: a participant that voted yes must hold its locks until it hears the outcome. If the coordinator dies, those rows stay locked.
  • Latency and availability: every transaction waits for the slowest participant, and is only available when all participants are.
  • Coupling: every service must speak the same protocol (XA) and expose its database to it. Kafka, Stripe, an email provider and most managed services can't take part at all.

The saga: local steps plus compensations

A saga splits the big transaction into a sequence of local transactions, one per service, each committed on its own. If a later step fails, the saga runs compensating transactions for the steps already done, in reverse order. A compensation is a semantic undo, not a rollback: you can't un-charge a card, but you can refund it; you can't delete an order a customer has seen, but you can cancel it.

An orchestrated saga, and compensations when a step fails

Two rules make sagas work:

  1. Every step and every compensation is idempotent and retryable. Messages are delivered at least once, and a timeout doesn't tell you whether the step happened. Retrying with the same idempotency key must be safe (see idempotency).
  2. Compensations must eventually succeed. Retry them with backoff; if one keeps failing, park it for a human. A saga that can't compensate leaves data inconsistent.

Order the steps wisely

Put steps that are easy to undo first and steps that can't be undone last. The step after which the saga can only go forward is called the pivot. A good checkout saga:

#StepCompensation
1Create order with status PENDINGMark order CANCELLED
2Reserve stock (a hold, not a deduction)Release the reservation
3Authorise the card (funds held, not taken)Void the authorisation
4Mark order CONFIRMED (pivot)None: from here, only forward
5Capture payment when shipping; send confirmation emailRetries until done

Authorising first and capturing later is how card payments are designed: a void is cleaner and cheaper than a refund, and the customer never sees a charge for a failed order.

Orchestration versus choreography

In orchestration, one component (the orchestrator) holds the saga's state and tells each service what to do next. In choreography, there is no central controller: each service listens for events and publishes its own.

Choreography: services react to each other's events

StyleGood atWeak atUse when
OrchestrationThe whole flow in one place; easy to see status, add steps, handle timeoutsThe orchestrator is another service to run; risk of putting business logic thereMore than 3 steps, branching, timeouts, anything you'll debug often
ChoreographyNo central component; services stay independentThe flow is spread across services; cycles and "who reacts to what" get hard to follow2–3 steps, simple flows, teams that own their events

Orchestrators are often built on durable workflow engines such as Temporal, AWS Step Functions or Camunda, which persist the saga's state after every step, so a crash resumes where it stopped rather than starting again.

Sagas give up isolation

While a saga runs, other requests can see its intermediate state: an order that is PENDING, stock that is reserved for an order that will be cancelled. A saga is atomic in the end (everything happens or is compensated) but not isolated. Countermeasures:

  • Semantic locks: a status such as PENDING that other operations respect ("can't ship a pending order").
  • Commutative updates: "reserve 1" and "release 1" give the same result in any order.
  • Re-read and check before acting on data another saga might be changing.

The dual-write problem and the transactional outbox

Inside each step hides a smaller problem. The Orders service must save the order and publish OrderCreated so the next step starts. Those are two different systems (the database and Kafka), so this is a dual write, and either can fail after the other succeeded.

The dual write: one of the two writes can be lost

The transactional outbox fixes it with the one transaction you do have: the service writes the order and an event row into an outbox table in the same local database transaction. A separate relay reads new outbox rows, publishes them to the broker, and marks them sent.

BEGIN;
INSERT INTO orders (id, customer_id, total, status) VALUES ('o-981', 'c-42', 59.90, 'PENDING');
INSERT INTO outbox (id, aggregate_id, type, payload)
  VALUES (gen_random_uuid(), 'o-981', 'OrderCreated', '{"order_id":"o-981","total":59.90}');
COMMIT;

The outbox, relayed by polling or by change data capture

Either both rows commit or neither does, so an event is published if and only if the order exists. The relay can crash after publishing but before marking a row sent, so it may publish twice: the outbox gives at-least-once delivery, and consumers must deduplicate by event ID (an inbox table of processed IDs is the mirror-image pattern).

Change data capture (CDC) is the other way to run the relay: a tool such as Debezium reads the database's own replication log (the Postgres WAL or the MySQL binlog) and turns committed changes into events. It adds no polling load, sees changes in commit order, and can even publish from tables that weren't designed with an outbox. Catch: it ties your events to table structure unless you still use an outbox table as the source, and it is another system to run.

Worked example: a checkout saga failing at step 3

  1. Orders writes o-981 PENDING plus an outbox event, in one transaction (~5 ms). The relay publishes OrderCreated (~50 ms later).
  2. The orchestrator asks Inventory to reserve 2 units, with idempotency key o-981:reserve. Reserved.
  3. It asks Payments to authorise €59.90 with key o-981:auth. The card is declined.
  4. Compensation in reverse: release the reservation (o-981:release), then mark the order CANCELLED. Each is retried until it succeeds.
  5. The customer sees "payment declined" within about a second. Total messages: around 8, each at least once, all idempotent.

Now a nastier case: at step 3 the call to the payment provider times out. You don't know if the card was authorised. Never assume: retry with the same idempotency key (the provider returns the original result if it already ran), or query the payment's status. Only then continue or compensate.

When not to use these tools

  • The data can live in one database. If two "services" share a lifecycle, a single local transaction is simpler and stronger. Splitting services along a transaction boundary is often a design mistake.
  • A single step is enough. One service calling a payment provider with an idempotency key needs no saga.
  • You need isolation, such as two transfers that must never see each other's partial state. Keep that inside one database (or a distributed SQL database that offers real transactions).
  • Small teams, small systems. A modular monolith with one database avoids the whole problem.

Which case studies use it

In the interview

Your turn: a travel booking saga

A travel site books a flight, a hotel and a rental car together, each through a different partner API. Flights are non-refundable after ticketing; hotels and cars can be cancelled free within 24 hours. The customer wants all three or nothing. Design the saga.

Compare with our answer
  • Orchestrated saga: several partners, timeouts and branching, so a central workflow with durable state.
  • Order the steps by reversibility: reserve the hotel and the car first (cancellable), then ticket the flight last as the pivot. Ideally hold the flight fare first (many airlines allow a short hold), and ticket only once the others are confirmed.
  • Compensations: cancel car, cancel hotel, release the flight hold. Each idempotent with a booking-specific key.
  • Payment: authorise the full amount at the start, capture after ticketing; void on failure.
  • Timeouts with partners: query the partner's booking status before retrying or compensating.
  • Failed compensations go to retries, then a human queue, with daily reconciliation against partner records.

A strong answer mentions: orchestration with durable state · the irreversible step last (the pivot) · compensations per step, idempotent · authorise then capture · handling unknown outcomes on timeout · a manual fallback and reconciliation.

Your turn: spot the bug

An Orders service does this:

def place_order(order):
    db.insert("orders", order)          # commits immediately
    kafka.publish("OrderCreated", order)

Sometimes orders exist that Payments never heard of; once, Payments charged for an order that doesn't exist in the database. Explain both and fix it.

Compare with our answer
  • Missing event: the service crashed (or Kafka was briefly unavailable) after the insert committed but before the publish. The order exists; no event.
  • Ghost order: an earlier version published first and inserted second; or a retry published, then the insert failed on a constraint. The event announced an order that never committed.
  • It's the dual-write problem: two systems, no shared transaction.

A strong answer mentions:

  • Write the order and an outbox row in one local transaction.
  • A relay or CDC (Debezium on the WAL) publishes outbox rows.
  • Delivery becomes at-least-once, so consumers deduplicate by event ID.
  • Never publish before the commit.
  • Monitor outbox lag (age of the oldest unsent row).
Transactions across services on one page
  • No single COMMIT across services; each step can fail independently, or time out with an unknown result.
  • 2PC: prepare then commit; blocks if the coordinator dies, needs XA everywhere. Fine inside distributed databases, avoided across services.
  • Saga: local transactions + compensations in reverse (refund, release, cancel). Atomic in the end, not isolated.
  • Rules: idempotent steps and compensations; compensations retried until done; durable saga state.
  • Order: reversible steps first, the irreversible pivot last; authorise then capture.
  • Orchestration (Temporal, Step Functions) for complex flows; choreography for 2–3 simple steps.
  • Isolation fixes: PENDING statuses (semantic locks), commutative updates, re-read before acting.
  • Outbox: business row + event row in one transaction; relay or CDC publishes; at-least-once, consumers dedup.
  • Timeouts: query status or retry with the same idempotency key; never guess.
Check what stuck

1. A team wants an order, its payment and its stock reservation, each in a different service's database, to commit atomically using two-phase commit. What is the main operational risk?

2. A checkout saga reserves stock, charges the card, then fails to create the shipment. What should happen next?

3. Orders saves an order and then publishes an event to Kafka. A crash between the two leaves orders that no other service knows about. What is the standard fix?

4. A payment authorisation call in a saga times out after 30 seconds. What should the orchestrator do?

5. A saga for travel bookings includes a non-refundable flight and cancellable hotel and car bookings. How should the steps be ordered?

6. An order saga has six steps, branching on payment method, and timeouts on two partners. Which approach fits best, and why?

7. With an outbox relay, consumers sometimes receive the same OrderCreated event twice. What is the correct response?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.