Transactions across services
How to keep data consistent when one action spans several services: why two-phase commit is avoided, sagas with compensations, the transactional outbox and change data capture.
On this page
- The problem: three services, one checkout
- Signals that you are facing it
- Two-phase commit, and why it's avoided
- The saga: local steps plus compensations
- The dual-write problem and the transactional outbox
- Worked example: a checkout saga failing at step 3
- When not to use these tools
- Which case studies use it
- In the interview
- Quiz
The problem: three services, one checkout
A shop has split its monolith into services, each with its own database: Orders, Payments and Inventory. Placing an order must create the order, charge the card and reserve the stock. In the monolith, that was one database transaction: either everything happened or nothing did.
Now there are three databases, and no single COMMIT covers them. Each step can fail independently: the card is declined after the order exists; the stock runs out after the card is charged; the Inventory service times out and you don't know whether it reserved anything. Without care you get customers charged for orders that don't exist, or orders shipped that were never paid for.
This lesson is about the tools for that situation: two-phase commit (and why it's usually avoided), sagas, the transactional outbox, and change data capture.
Signals that you are facing it
- One user action changes data owned by two or more services: order + payment + stock, booking + payment, ride + driver status.
- A database write plus a message: "save the order and publish OrderCreated".
- Calls to external systems that can't join your transaction: payment providers, email, shipping partners.
- Requirements like "never charge without an order" or "release the seat if payment fails".
Say early: "This spans services, so there's no single transaction. I'll use a saga with compensating steps, and an outbox so each service's database change and its event can't diverge."
Two-phase commit, and why it's avoided
Two-phase commit (2PC) is the classic way to make several databases commit atomically. A coordinator asks every participant to prepare (write the change durably, lock the rows, promise it can commit), and only if all vote yes does it tell them to commit.
Two-phase commit, and the coordinator that blocks everyone
It works, and it is used: inside distributed databases such as Spanner and CockroachDB, where each participant and the coordinator's state are themselves replicated with consensus, so a single crashed machine doesn't block anyone for long. Across independent services, it is avoided because:
- Blocking: a participant that voted yes must hold its locks until it hears the outcome. If the coordinator dies, those rows stay locked.
- Latency and availability: every transaction waits for the slowest participant, and is only available when all participants are.
- Coupling: every service must speak the same protocol (XA) and expose its database to it. Kafka, Stripe, an email provider and most managed services can't take part at all.
The saga: local steps plus compensations
A saga splits the big transaction into a sequence of local transactions, one per service, each committed on its own. If a later step fails, the saga runs compensating transactions for the steps already done, in reverse order. A compensation is a semantic undo, not a rollback: you can't un-charge a card, but you can refund it; you can't delete an order a customer has seen, but you can cancel it.
An orchestrated saga, and compensations when a step fails
Two rules make sagas work:
- Every step and every compensation is idempotent and retryable. Messages are delivered at least once, and a timeout doesn't tell you whether the step happened. Retrying with the same idempotency key must be safe (see idempotency).
- Compensations must eventually succeed. Retry them with backoff; if one keeps failing, park it for a human. A saga that can't compensate leaves data inconsistent.
Order the steps wisely
Put steps that are easy to undo first and steps that can't be undone last. The step after which the saga can only go forward is called the pivot. A good checkout saga:
| # | Step | Compensation |
|---|---|---|
| 1 | Create order with status PENDING | Mark order CANCELLED |
| 2 | Reserve stock (a hold, not a deduction) | Release the reservation |
| 3 | Authorise the card (funds held, not taken) | Void the authorisation |
| 4 | Mark order CONFIRMED (pivot) | None: from here, only forward |
| 5 | Capture payment when shipping; send confirmation email | Retries until done |
Authorising first and capturing later is how card payments are designed: a void is cleaner and cheaper than a refund, and the customer never sees a charge for a failed order.
Orchestration versus choreography
In orchestration, one component (the orchestrator) holds the saga's state and tells each service what to do next. In choreography, there is no central controller: each service listens for events and publishes its own.
Choreography: services react to each other's events
| Style | Good at | Weak at | Use when |
|---|---|---|---|
| Orchestration | The whole flow in one place; easy to see status, add steps, handle timeouts | The orchestrator is another service to run; risk of putting business logic there | More than 3 steps, branching, timeouts, anything you'll debug often |
| Choreography | No central component; services stay independent | The flow is spread across services; cycles and "who reacts to what" get hard to follow | 2–3 steps, simple flows, teams that own their events |
Orchestrators are often built on durable workflow engines such as Temporal, AWS Step Functions or Camunda, which persist the saga's state after every step, so a crash resumes where it stopped rather than starting again.
Sagas give up isolation
While a saga runs, other requests can see its intermediate state: an order that is PENDING, stock that is reserved for an order that will be cancelled. A saga is atomic in the end (everything happens or is compensated) but not isolated. Countermeasures:
- Semantic locks: a status such as
PENDINGthat other operations respect ("can't ship a pending order"). - Commutative updates: "reserve 1" and "release 1" give the same result in any order.
- Re-read and check before acting on data another saga might be changing.
The dual-write problem and the transactional outbox
Inside each step hides a smaller problem. The Orders service must save the order and publish OrderCreated so the next step starts. Those are two different systems (the database and Kafka), so this is a dual write, and either can fail after the other succeeded.
The dual write: one of the two writes can be lost
The transactional outbox fixes it with the one transaction you do have: the service writes the order and an event row into an outbox table in the same local database transaction. A separate relay reads new outbox rows, publishes them to the broker, and marks them sent.
BEGIN;
INSERT INTO orders (id, customer_id, total, status) VALUES ('o-981', 'c-42', 59.90, 'PENDING');
INSERT INTO outbox (id, aggregate_id, type, payload)
VALUES (gen_random_uuid(), 'o-981', 'OrderCreated', '{"order_id":"o-981","total":59.90}');
COMMIT;
The outbox, relayed by polling or by change data capture
Either both rows commit or neither does, so an event is published if and only if the order exists. The relay can crash after publishing but before marking a row sent, so it may publish twice: the outbox gives at-least-once delivery, and consumers must deduplicate by event ID (an inbox table of processed IDs is the mirror-image pattern).
Change data capture (CDC) is the other way to run the relay: a tool such as Debezium reads the database's own replication log (the Postgres WAL or the MySQL binlog) and turns committed changes into events. It adds no polling load, sees changes in commit order, and can even publish from tables that weren't designed with an outbox. Catch: it ties your events to table structure unless you still use an outbox table as the source, and it is another system to run.
Worked example: a checkout saga failing at step 3
- Orders writes
o-981 PENDINGplus an outbox event, in one transaction (~5 ms). The relay publishesOrderCreated(~50 ms later). - The orchestrator asks Inventory to reserve 2 units, with idempotency key
o-981:reserve. Reserved. - It asks Payments to authorise €59.90 with key
o-981:auth. The card is declined. - Compensation in reverse: release the reservation (
o-981:release), then mark the orderCANCELLED. Each is retried until it succeeds. - The customer sees "payment declined" within about a second. Total messages: around 8, each at least once, all idempotent.
Now a nastier case: at step 3 the call to the payment provider times out. You don't know if the card was authorised. Never assume: retry with the same idempotency key (the provider returns the original result if it already ran), or query the payment's status. Only then continue or compensate.
When not to use these tools
- The data can live in one database. If two "services" share a lifecycle, a single local transaction is simpler and stronger. Splitting services along a transaction boundary is often a design mistake.
- A single step is enough. One service calling a payment provider with an idempotency key needs no saga.
- You need isolation, such as two transfers that must never see each other's partial state. Keep that inside one database (or a distributed SQL database that offers real transactions).
- Small teams, small systems. A modular monolith with one database avoids the whole problem.
Which case studies use it
- Design Amazon checkout: the order, payment and inventory saga.
- Design a payment system: idempotency keys, ledgers, reconciliation, outbox events.
- Design food delivery: order, restaurant acceptance, courier assignment and payment as a saga.
- Design Ticketmaster: seat hold, payment, confirmation, with release as compensation.
- Design Uber: trip, driver assignment and payment across services.
- Related: contention, message queues, consistency and CAP.
In the interview
A travel site books a flight, a hotel and a rental car together, each through a different partner API. Flights are non-refundable after ticketing; hotels and cars can be cancelled free within 24 hours. The customer wants all three or nothing. Design the saga.
Compare with our answer
- Orchestrated saga: several partners, timeouts and branching, so a central workflow with durable state.
- Order the steps by reversibility: reserve the hotel and the car first (cancellable), then ticket the flight last as the pivot. Ideally hold the flight fare first (many airlines allow a short hold), and ticket only once the others are confirmed.
- Compensations: cancel car, cancel hotel, release the flight hold. Each idempotent with a booking-specific key.
- Payment: authorise the full amount at the start, capture after ticketing; void on failure.
- Timeouts with partners: query the partner's booking status before retrying or compensating.
- Failed compensations go to retries, then a human queue, with daily reconciliation against partner records.
A strong answer mentions: orchestration with durable state · the irreversible step last (the pivot) · compensations per step, idempotent · authorise then capture · handling unknown outcomes on timeout · a manual fallback and reconciliation.
An Orders service does this:
def place_order(order):
db.insert("orders", order) # commits immediately
kafka.publish("OrderCreated", order)
Sometimes orders exist that Payments never heard of; once, Payments charged for an order that doesn't exist in the database. Explain both and fix it.
Compare with our answer
- Missing event: the service crashed (or Kafka was briefly unavailable) after the insert committed but before the publish. The order exists; no event.
- Ghost order: an earlier version published first and inserted second; or a retry published, then the insert failed on a constraint. The event announced an order that never committed.
- It's the dual-write problem: two systems, no shared transaction.
A strong answer mentions:
- Write the order and an outbox row in one local transaction.
- A relay or CDC (Debezium on the WAL) publishes outbox rows.
- Delivery becomes at-least-once, so consumers deduplicate by event ID.
- Never publish before the commit.
- Monitor outbox lag (age of the oldest unsent row).
- No single
COMMITacross services; each step can fail independently, or time out with an unknown result. - 2PC: prepare then commit; blocks if the coordinator dies, needs XA everywhere. Fine inside distributed databases, avoided across services.
- Saga: local transactions + compensations in reverse (refund, release, cancel). Atomic in the end, not isolated.
- Rules: idempotent steps and compensations; compensations retried until done; durable saga state.
- Order: reversible steps first, the irreversible pivot last; authorise then capture.
- Orchestration (Temporal, Step Functions) for complex flows; choreography for 2–3 simple steps.
- Isolation fixes: PENDING statuses (semantic locks), commutative updates, re-read before acting.
- Outbox: business row + event row in one transaction; relay or CDC publishes; at-least-once, consumers dedup.
- Timeouts: query status or retry with the same idempotency key; never guess.
1. A team wants an order, its payment and its stock reservation, each in a different service's database, to commit atomically using two-phase commit. What is the main operational risk?
Participants that voted yes cannot decide alone, so their rows stay locked while the coordinator is down. That blocking, plus the latency and protocol coupling, is why services use sagas instead.
2. A checkout saga reserves stock, charges the card, then fails to create the shipment. What should happen next?
Each step committed locally, so it is undone by a semantic compensation, newest first. There is no cross-service rollback, and deleting records loses the audit trail.
3. Orders saves an order and then publishes an event to Kafka. A crash between the two leaves orders that no other service knows about. What is the standard fix?
The outbox turns the dual write into one local transaction; a relay or CDC then publishes. Publishing first creates ghost events; Kafka transactions don't include your database.
4. A payment authorisation call in a saga times out after 30 seconds. What should the orchestrator do?
A timeout means the outcome is unknown. Retrying with the same key returns the original result if it ran; a fresh key risks charging twice.
5. A saga for travel bookings includes a non-refundable flight and cancellable hotel and car bookings. How should the steps be ordered?
Put reversible steps before the irreversible pivot. If the hotel or car fails, they are cancelled and no flight was ticketed. Parallel steps could ticket the flight before a failure is known.
6. An order saga has six steps, branching on payment method, and timeouts on two partners. Which approach fits best, and why?
Complex sagas are easier to follow, debug and change with an orchestrator, usually on a durable workflow engine. Choreography suits two or three simple steps.
7. With an outbox relay, consumers sometimes receive the same OrderCreated event twice. What is the correct response?
The relay can crash after publishing but before marking the row sent. Deleting first risks losing events, and CDC is also at-least-once. Idempotent consumers are the fix.
Score 60% or more and the lesson is marked complete.