An order touches the payment, inventory and shipping services. Would you use two-phase commit or a saga?

A saga, in most cases. Two-phase commit makes every participant hold locks from the vote until the decision, a crashed coordinator can leave them blocked, and many services and brokers do not support it. A saga commits each step on its own and undoes earlier steps with compensations if a later one fails. The trade-off is that other users can see a half-finished order, so the design needs pending statuses.

What happens if a compensation itself fails, for example the refund call times out?

Retry it until it succeeds, and make it idempotent with a key so a repeat cannot refund twice. The orchestrator saves progress, so a restart continues the compensation. Put a retry limit and an alert on it, and send stuck sagas to a person. The trade-off is that cancelling is eventual, not instant.

A service saves an order and publishes an "order placed" event. How do you stop it losing the event if it crashes in between?

Use the outbox pattern. Write the order and the event row to the same database in one local transaction, then have a relay read unsent rows and publish them. A crash cannot separate the two writes. The relay may publish an event twice if it crashes before marking it sent, so consumers must be idempotent. The trade-off is a small delay and a relay to run.

A client times out calling your payment API and retries. How do you avoid charging twice?

The client sends a unique idempotency key with the request. The server stores the key with the result in the same transaction as the change it protects (for a charge that goes to an outside provider, also pass the key on so that the provider removes duplicates), and on a retry with the same key it returns the saved result instead of charging again. The trade-off is storing keys and choosing how long to keep them.

Design the booking flow for concert tickets so a seat is never sold twice and a failed payment frees the seat.

Reserve the seat with an expiry, charge the card with an idempotency key, then confirm the seat. If the payment fails or the hold expires, release the seat as the compensation. An orchestrator saves each step so a crash resumes, and events go out through an outbox. The trade-off is that a seat shows as held for a few minutes even if the buyer leaves.

When would you choose two-phase commit over a saga?

Choose it when the participants are a few databases that you control and that support it (for example PostgreSQL prepared transactions, once enabled, or MySQL XA), the transaction is short, and you need all or nothing without a long-lived half-finished state. The trade-off is that locks are held from the vote until the decision and a coordinator crash can block everyone, so keep the transaction short and make the coordinator highly available.

Distributed Transactions | TheLearningLoop

Why you need distributed transactions, and when to use them

In one database, a transaction makes several changes succeed or fail together. Once the data is split across services (payment, inventory, shipping), each with its own database, no single database can do that for you. A crash halfway through leaves a card charged and no parcel booked.

A distributed transaction is any technique that keeps changes in several systems consistent despite crashes and retries. It gives you two things interviewers always ask about:

Consistency: the order is either fully done or fully cancelled, not stuck in between for ever.
Safe retries: when a request or message is repeated, the effect happens once.

Four words recur below. A lock stops other work from touching a row until its owner finishes. A participant is one service or database taking part in the transaction. Atomic means all or nothing. An operation is idempotent when doing it twice has the same effect as doing it once, such as setting an order's status to "paid".

Use one when:

A single business action must change data owned by different services or databases.
A partial result has real cost: money taken without a purchase, stock reserved for an order that never exists.
Messages can be lost or repeated, which is true of almost every network.

Skip it, or use something else, when:

The data can live in one database. A normal local transaction is simpler and stronger.
A short period of disagreement is acceptable and can be fixed later, such as a view counter.
The steps are independent. Two emails need no transaction between them.
You are splitting services only for fashion. Merging two services that always change together is often the best fix.

Try it

Place the same order with two approaches, then break something. Each service has its own database and cannot see the others.

Distributed transaction

Step 1 of 5 · Press Space to play, arrow keys to step

ApproachFailure

Initial state. One order must charge the card, reserve stock and book shipping. Each service has its own database.

tick:: 0
messages:: 0
locked service-ticks:: 0
outcome:: in progress

1 tick = one step. Locked service-ticks adds up how long services sat holding locks.

Speed

Things to try:

Run two-phase commit with no failure. All three services lock, vote yes and commit together. The order is never left half-finished, because the commit is all or nothing.
Run a saga with "Shipping is unavailable". The payment and stock are already committed, so the saga has to refund and release them, in reverse order.
Run two-phase commit with "Coordinator crashes mid-way". All three services sit locked and cannot decide alone. Then run the same failure with a saga and note that nothing is locked.
Run two-phase commit with "Inventory has no stock". Payment is locked when Inventory votes no, and the abort releases it. Shipping is never asked in this run, although a real coordinator often sends the prepare requests to all participants at once.
Watch the counters. Locked service-ticks adds up how long services sat holding locks: it is above zero for every two-phase commit run and stays at 0 for a saga. Messages counts requests and replies in this simulation and leaves out the acknowledgements a real system also sends, so do not read it as a cost comparison.

Approaches to a distributed transaction

Approach	How it works	Good when	Weakness
Two-phase commit (2PC)	A coordinator asks everyone to prepare, then tells all to commit or abort	A few databases that support it, short transactions, strict all-or-nothing	Locks are held while waiting, and a coordinator crash can block everyone
Saga (choreography)	Each service does its step and publishes an event that triggers the next. Failures trigger compensating events	Few steps, loosely coupled teams	The flow is spread across services and is hard to follow and debug
Saga (orchestration)	One orchestrator calls each step in turn and runs the compensations if one fails	Many steps, branching logic	The orchestrator is one more component to run and keep reliable
Outbox pattern	Write the data and the event to the same database in one local transaction. A relay publishes the event later	You must update your database and send a message	Events can be sent twice, so consumers must be idempotent
Idempotency keys	The caller attaches a unique key. The server remembers it and returns the saved result on a retry	Any request that can be retried, such as payments	You need to store keys and decide how long to keep them

Real systems usually combine them: a saga for the business flow, the outbox for reliable events inside each step, and idempotency keys on every step.

Two-phase commit vs sagas

2PC gives atomic commit: every participant commits or none does, so there is no long-lived half-finished order. Readers can still briefly see one participant committed before another while the commit messages arrive, unless the databases also coordinate their reads. The price is that every participant holds locks from the vote until the decision, and a crashed coordinator can leave them blocked. It also needs every participant to support the protocol, which many APIs and message brokers do not.
A saga gives up that isolation. Each step commits straight away, so other users can see a half-finished order. It never holds locks across services and survives a coordinator restart, because progress is saved in a log. The price is that you must write a compensation for every step.

Two more details about 2PC are worth knowing. A participant may still abort on its own before it votes yes, for example when its local checks fail or a timeout expires. After it votes yes it must wait for the decision. The coordinator writes its decision to a log, so after a restart it can finish the job, and the block lasts only until then. Some systems shorten that wait by replicating the coordinator: Google Spanner, for example, runs two-phase commit across groups of replicas that use Paxos, so the failure of one machine does not stall the decision.

How compensations and retries work

A compensation is a new action that semantically undoes a committed step: a refund, not a deleted charge. Plan for these points:

Run them in reverse order of the steps that completed.
Make every step and compensation idempotent. A retry after a timeout must not charge or refund twice.
Some steps cannot be undone, such as an email that is already sent. Put those last, or make them the point of no return.
Compensations can fail too. Retry them until they succeed and alert a person if they cannot.
Save progress before and after each step, so a restarted orchestrator carries on instead of starting again.
Order the steps with care. Put steps that are cheap to undo first and steps that cannot be undone last (see point 3). If a step is often refused, such as a stock check, consider running it before the steps that move money, so the saga rarely has to refund anything. The order used in the simulation and examples here was chosen to show compensations, not as the best order.

Distributed transactions vs the alternatives

Concern	Saga + outbox	Two-phase commit	One database
Works across services you do not control	Yes	Only if all support it	No
Other users never see a partial result	No	Yes, apart from a brief window while commits arrive	Yes
Holds locks while waiting on other services	No	Yes	Only inside one database
Survives a coordinator crash	Yes, from its log	Can block	Not needed
Extra code to write	Compensations	Little	None
Best for	Business flows across services	A few cooperating databases	Everything that can fit in one

Sagas win when services are separate and long-running. If the data can sit in one database, that is simpler and stronger than either.

Avoiding a new single point of failure

The coordinator or orchestrator now sits in the middle of every order. Protect it:

Persist its state (which steps are done) in a database, so any instance can resume after a crash.
Run several instances and let only one drive a given saga at a time.
Set timeouts and retry limits on each step, and move stuck sagas to an alert instead of retrying for ever.
With 2PC, keep transactions short and the coordinator highly available, because a crash while participants are prepared blocks them.

You accept more moving parts and, with sagas, a short window where the data disagrees.

Common mistakes

Retrying without idempotency. A timeout does not mean the request failed. Add an idempotency key so a repeat is safe.
Writing to the database and publishing an event separately. A crash between the two loses the event or sends one for data that never committed. Use the outbox.
No compensation for a step. Decide the undo before you write the step.
Treating the saga as isolated. Other users can read the middle state. Show "pending" statuses and avoid acting on half-finished data.
Holding 2PC locks across slow calls. Keep prepared transactions short.
Retrying for ever. Cap retries, then alert a person.

How to talk about distributed transactions in an interview

Say first that you would avoid them by keeping related data in one service and database where you can.
If the flow crosses services, pick a saga with an orchestrator, name the compensation for each step, and say why you avoid 2PC (locks and blocking).
Mention idempotency keys and the outbox unprompted. They show you know that messages repeat and that "write then publish" is not atomic.
Name the trade-off: a saga gives up isolation, so users may see a pending state, in exchange for availability and no cross-service locks.

Where you meet distributed transactions in real systems

Any purchase, booking or transfer that touches more than one service needs one. Here are three drawn out, the first with SQL you can run, then more examples.

Example 1: Placing an order safely

The order service must save the order and tell other services about it. If it saves and then crashes before publishing, the event is lost. The outbox pattern writes the order and the event in one local transaction, and a separate relay publishes the event afterwards. An idempotency key stops a retried request from creating a second order. Open the example below to see the picture and the SQL.

Config: an outbox table and an idempotency keySetup 3 tables and a relay queryCode no code changes

The order and its event are saved together in one local transaction. A relay reads unsent events and publishes them. If the relay crashes after publishing, the event is sent again, so consumers must be idempotent.

-- PostgreSQL: the outbox pattern plus an idempotency key, in one local transaction.
CREATE TABLE orders (
  id         bigint PRIMARY KEY,
  total      numeric(10,2) NOT NULL,
  status     text NOT NULL
);

-- The outbox: events waiting to be published. Written in the SAME transaction as the order.
CREATE TABLE outbox (
  id           bigserial PRIMARY KEY,
  topic        text NOT NULL,
  payload      jsonb NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz            -- NULL means not sent yet
);

-- Idempotency keys: the client sends the same key when it retries, so a retry cannot charge twice.
CREATE TABLE idempotency_keys (
  key        text PRIMARY KEY,
  response   jsonb NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);

-- Place an order. Either all three rows are written or none are.
BEGIN;
INSERT INTO idempotency_keys (key, response)
VALUES ('order-req-7f3a', '{"order_id": 1001, "status": "placed"}');
INSERT INTO orders (id, total, status) VALUES (1001, 750.00, 'placed');
INSERT INTO outbox (topic, payload)
VALUES ('order.placed', '{"order_id": 1001, "total": 750.00}');
COMMIT;

-- A retry with the same key fails on the primary key instead of creating a second order.
-- ON CONFLICT DO NOTHING lets the app notice that and return the saved response.
INSERT INTO idempotency_keys (key, response)
VALUES ('order-req-7f3a', '{}')
ON CONFLICT (key) DO NOTHING;

-- A relay process polls for unsent events. SKIP LOCKED lets several relays run without clashing.
BEGIN;
SELECT id, topic, payload FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 10
FOR UPDATE SKIP LOCKED;
-- ...publish each event to the broker, then mark it as sent (replace 1 with the ids you published):
UPDATE outbox SET published_at = now() WHERE id IN (1);
COMMIT;
-- If the relay crashes after publishing but before this UPDATE, the event is sent again.
-- That is fine only because consumers are idempotent.

Both scripts were run against real servers (PostgreSQL 16 and MySQL 8): creating the tables, placing an order, repeating the same idempotency key, and reading and marking the outbox row with SKIP LOCKED. Names and sizes are examples.

Two limits of the relay are worth knowing. It holds its row locks while it publishes, so keep batches small. With several relays the global order of events is not guaranteed, so if order matters, send the events for one order through a single relay or a broker partition keyed by the order id, and let consumers cope with events arriving out of order.

Example 2: A booking saga with compensations

Booking a trip means charging a card, reserving a seat and booking a hotel, and three services own those. An orchestrator runs the steps in order. If the hotel fails, it refunds the card and releases the seat.

Each step commits straight away. When the hotel booking fails, the orchestrator runs the compensations for the earlier steps in reverse order.

Example 3: Two-phase commit and a crashed coordinator

Two databases that support 2PC can commit a transfer together. The risky moment is after every participant has voted yes and before the decision arrives.

Both databases voted yes and hold locks. The coordinator crashed before sending the decision, so neither may commit or abort alone until it returns.

More real-world scenarios

These scenarios are illustrative patterns, not descriptions of any one company's system.

4. Online shopping checkout

Payment, stock and delivery are separate services. A saga charges the card, reserves stock and books delivery, and refunds if any later step fails.

5. Bank transfers between accounts

Two accounts in one bank database use an ordinary transaction. Transfers between banks cannot, so each side records the movement and the pair is reconciled, with reversals when something fails.

6. Flight and hotel booking

Seats and rooms are held for a short time (a reservation with an expiry) and confirmed on payment. The expiry is the compensation if the customer never pays.

7. Ride-hailing

Matching a driver, starting a trip and charging the rider are separate steps. If the charge fails, the trip is ended and the driver freed.

8. Sending events from a database

A service that saves a row and publishes a message to a broker uses the outbox, so the message is sent if and only if the row was saved.

9. Payment APIs

A client that times out cannot know whether the charge happened. It retries with the same idempotency key, and the provider returns the first result instead of charging again.

10. Microservice data migrations

Moving a customer between two databases is done as a saga: copy, switch, verify, delete, with a way back at each step.

Tools and managed services

Kind	Name	Notes
Workflow engine	Temporal	Runs sagas as durable code that survives crashes
Workflow engine	Camunda	Models long-running business processes
Managed workflows	AWS Step Functions	State machines that can call compensation steps
Managed workflows	Azure Durable Functions	Orchestrations written as code
Message broker	Apache Kafka, RabbitMQ	Carry saga and outbox events
Change data capture	Debezium	Reads the database log and publishes outbox rows
Database	PostgreSQL, MySQL	Support prepared transactions for 2PC and `SKIP LOCKED` for relays

The takeaway

Whenever one action must change data in several services, avoid a shared transaction: use a saga with compensations, write events through an outbox, and put an idempotency key on every retryable step.

Distributed Transactions

Key takeaways