All or Nothing: Distributed Transactions, Explained Like You're New

In a lot of software, checkout is the most dangerous button. One click has to set off two things that must both happen or both not happen: the payment service charges the card, and the inventory service sets the item aside. If the charge goes through and the reservation fails, a customer is holding a receipt for nothing. If the reservation goes through and the charge fails, a warehouse is holding a box for nobody. Neither middle state is acceptable, and yet the two halves live in different services with different databases.

This post is about how backends try to get separate machines to promise "all or nothing," why a perfect version of that promise can't exist, and what working systems do instead. If you're new, Sections 1 and 2 assume no background, every term is defined the first time it appears, and each section has diagrams with a note on how to read them. Sections 3 through 6 are the machinery: two-phase commit, the impossibility results behind it, sagas, and the transactional outbox. Section 7 covers how each one fails in practice, Section 8 has the numbers and a decision framework you can use on a whiteboard, and Section 9 is the principal-level answer: draw your boundaries so you rarely need any of this. The one-page summary is at the end, under All or nothing, distilled.

---

Section 1: What a transaction is, inside one database

A transaction is a group of reads and writes that a database treats as one unit. You open it with BEGIN, make your changes, and finish with COMMIT (keep everything) or ROLLBACK (keep nothing). The guarantees go by the acronym ACID:

• Atomicity: all of the transaction's changes happen, or none do. A transfer debits one account and credits the other, never just one of the two.
• Consistency: the data moves from one valid state to another, with the rules you declared, like "balance can't go negative" or "email must be unique," still holding afterward.
• Isolation: transactions running at the same time don't see each other's half-finished work. Databases offer several isolation levels that trade strictness for speed, and the next post (#16) walks through them.
• Durability: once COMMIT returns, the change survives a crash or power loss.

One database can promise all this because one machine controls everything that matters. It keeps a write-ahead log, a file where every change is recorded before it's applied, so after a crash it can replay finished work and undo unfinished work. It holds the locks, markers that stop other transactions from changing (or sometimes reading) a row until the current one finishes, that keep concurrent transactions apart. And it owns the single moment, the log record for the COMMIT, when a transaction becomes final.

Figure 1. A transfer inside one database. The important line is step 7: the transaction becomes final when its COMMIT record reaches the write-ahead log on disk, and everything after that (updating table pages, releasing locks) can be redone from the log after a crash. One machine owns the log, the locks, and that instant. Distributed transactions are hard because no single machine owns all three.

This post is about what happens when you leave that one machine. Split checkout into a payment service with its own database and an inventory service with its own database, and each one can still run a perfect local transaction. Neither can see the other's. There's no shared BEGIN or COMMIT, and no single log to consult when something dies halfway through. Every technique below is an attempt to rebuild some part of ACID across that gap.

---

Section 2: Why "both or neither" is hard across machines

Walk through it. A request arrives: charge $50, reserve one unit. You tell the payment service to charge, and it replies that the charge committed. You tell the inventory service to reserve, and the call times out. Did the reservation happen? The service might have reserved the item and lost the reply on the way back, or it might have crashed before doing anything. From the caller's side, those two cases look identical. That ambiguity, not being able to tell "it failed" from "it worked and the answer got lost," is the whole difficulty.

Figure 2. The core ambiguity. After the timeout, the checkout service has exactly the same information in all three worlds, yet the right action differs: retry in world 1, do nothing in world 2, wait in world 3. Every technique in this post is a way of acting safely without knowing which world you're in.

Both obvious reactions are wrong. Retrying blindly is how a card gets charged twice, which is the reason idempotency (#6) exists: an operation is idempotent if doing it twice has the same effect as doing it once. Giving up is how you keep the money and ship nothing.

And two services is the easy case. Each participant can end up committed or not, so with n participants there are 2^n combinations of outcomes, and only two of them (everyone committed, nobody committed) are acceptable. Two services leave 2 bad combinations. Three leave 6. Five leave 30. Every service you add to a business operation multiplies the partial states you have to detect and fix.

Figure 3. Why "just add one more service" is expensive. Each participant either committed or didn't, so n participants have 2^n combinations, and all but two of them are inconsistent. The growth is exponential: six services leave 62 mixed states, and your recovery logic has to handle every one that can actually happen.

Compare that with Section 1. Inside one database, failure is unambiguous: the log says exactly what committed. Across machines, each one knows its own state and nobody knows the global state. Everything below is a way of managing that ambiguity, because, as Section 4 shows, nothing can make it go away.

---

Section 3: Two-phase commit, the first serious attempt

Two-phase commit, usually written 2PC, is the textbook answer. It's worth understanding even if you never run it, because every other approach in this post is a reaction to its weaknesses.

2PC adds a coordinator, a process that runs the commit on everyone's behalf. When the checkout is ready to commit, the coordinator runs two phases:

1. Prepare. The coordinator asks each participant, "Can you commit?" Each participant does all the work short of committing, makes sure nothing can stop it later (constraints checked, locks held, changes written to disk), records "prepared" durably in its own log, and votes yes. Or it votes no.
2. Commit. If every vote is yes, the coordinator writes its decision, COMMIT, durably to its own log, and only then tells every participant to commit. If any vote is no, or a participant doesn't answer in time, the decision is ABORT and everyone rolls back.

Figure 4. Two-phase commit when nothing fails. Count the costs: two round trips to each participant, three durable log writes on the critical path (two PREPARED records and the coordinator's decision), and locks held from step 2 until the last acknowledgment. Step 7 is the single moment of commit, the distributed equivalent of the COMMIT record in Figure 1.

Two details make it work. A participant that voted yes has given up the right to change its mind: from then on it must do whatever the coordinator decides. And the decision becomes final the moment it reaches the coordinator's log, before any participant hears about it. That log write is the single moment the distributed transaction commits, which is how 2PC recreates the "one moment where commit becomes final" from Section 1.

The price comes in two parts. The first is latency: at least two network round trips to every participant, plus durable log writes on the critical path, with locks held the whole time. Martin Kleppmann's Designing Data-Intensive Applications cites reports of distributed transactions in MySQL running more than ten times slower than single-node ones.

The second part is worse, and it has a name: blocking. If the coordinator crashes after participants have voted yes but before they hear the decision, those participants are in doubt. They can't abort on their own, because the coordinator may already have committed and told someone else. They can't commit on their own either. So they wait, holding their locks, while every query that needs those rows queues behind them, until the coordinator comes back.

Figure 5. The blocking problem. Between step 5 and the coordinator's recovery, both participants hold locks they can't release on their own, and unrelated queries pile up behind them. The participants aren't broken; they're obeying the protocol, because only the coordinator's log knows the decision.

You can see how seriously databases take this. PostgreSQL's PREPARE TRANSACTION command, its building block for 2PC, exists for external transaction managers, and the documentation says that unless you have one, you should leave the feature disabled, with max_prepared_transactions at its default of zero. A forgotten prepared transaction keeps holding its locks and stops VACUUM, Postgres's background cleanup of old row versions, from reclaiming storage, and in extreme cases the database shuts down to protect itself. The cross-vendor standard for 2PC is X/Open XA, which MySQL and most commercial databases implement.

A natural question is whether adding a third phase fixes blocking. Three-phase commit (3PC) tries, by adding a "pre-commit" round so participants can finish on their own, but it assumes network delays are bounded and failures are detectable, and on real networks a partition can still lead it to inconsistent decisions. It's rarely used in practice. So 2PC across independently deployed services fell out of favor, but 2PC inside a distributed database is alive and well. Google's Spanner uses it with one crucial change: each participant and the coordinator are replicated groups of machines that agree on their state through a consensus protocol (Paxos, in Spanner's case). If the coordinator's leader machine dies, another replica already holds the logged decision and finishes the job, so one crash no longer leaves anyone in doubt. Jim Gray and Leslie Lamport formalized this as Paxos Commit and showed that classic 2PC is its special case with a single, unreplicated coordinator. The blocking problem comes from a fragile coordinator, and there are two ways out: replicate the coordinator, or design so you don't need one.

Figure 6. How Spanner-style databases keep 2PC without the blocking. The protocol between the groups is still two-phase commit, but each box is now a group of machines that agree through consensus (Paxos or Raft). A crashed coordinator becomes a leader election instead of an outage, which is what Gray and Lamport's Paxos Commit formalizes.

---

Section 4: The two generals, and what "impossible" actually means

Two generals are camped on opposite hills with an enemy in the valley between them. They win only if they attack at the same time, and they can communicate only by messenger through the valley, where messengers can be captured. General A sends "attack at dawn." Did B get it? B sends back "confirmed." Did A get the confirmation? If it was captured, A might stay home, so B can't be sure either. A confirms the confirmation, and the same problem moves up one level. However many messages they exchange, whoever sent the last one can't know it arrived. No finite exchange lets both generals be certain they'll act together.

Figure 7. The two generals. Each acknowledgment needs its own acknowledgment, forever. Replace the generals with services and the messengers with network packets, and this is the timeout in Figure 2: no amount of extra messaging turns "probably" into "certainly."

The puzzle was first described by Akkoyunlu, Ekanadham, and Huber in 1975, and Jim Gray gave it the generals story in 1978. Swap the generals for your payment and inventory services, the messengers for network packets, and capture for timeouts and crashes, and Section 2's ambiguity becomes a theorem.

What it rules out is narrower than people often claim, and the difference matters. "Distributed agreement is impossible" is a misreading. 2PC does guarantee that every participant that decides reaches the same decision, and that safety property holds whatever the network does. What no protocol can also guarantee is that everyone eventually decides when machines can crash and messages can be arbitrarily slow. Two separate results sit behind this, and it helps to keep them apart. The two generals problem is about unreliable links: even with no crashes at all, lost messages mean nobody can be certain the other side knows. The second result is about crashes: Fischer, Lynch, and Paterson proved in 1985, in a result usually called FLP, that in a fully asynchronous system, where there's no upper bound on how long a message can take, no deterministic protocol can guarantee that consensus will ever be reached if even one process might crash, even when every message is eventually delivered. Atomic commit is a close cousin of consensus with one extra rule (any single "no" vote forces an abort), and it inherits the same limit.

So every real protocol picks what to give up while a failure is happening. 2PC keeps agreement and gives up progress: it waits. Consensus protocols like Paxos and Raft keep agreement at all times and make progress whenever the network behaves well enough for timeouts to work, which in practice is nearly always. That reframes the engineering question. You can't close the window of uncertainty, but you can choose what happens inside it: wait until you know (2PC), or act now and repair later (everything from Section 5 on).

Figure 8. The only real choice. Neither branch removes the uncertainty; each decides who pays for it. Waiting makes the system unavailable during failures, acting makes intermediate states visible, and the repair machinery on the bottom right is what makes the second branch safe.

The repair side has a name: reconciliation. It means timeouts, retries made safe by idempotency (#6), and periodic jobs that compare what the payment service thinks happened with what the inventory service thinks happened, then fix the difference or flag it for a person.

The same reasoning settles arguments about "exactly-once delivery." Over an unreliable network, a sender that times out can't know whether its message arrived, so it either sends again (at-least-once, with possible duplicates) or doesn't (at-most-once, with possible loss). What you can build is an exactly-once effect: at-least-once delivery to a receiver that ignores duplicates. Kafka's exactly-once feature is real, and it works by keeping the reads, the writes, and the record of progress inside Kafka's own transactional log. The moment a result leaves Kafka for another system, that system has to cooperate by handling repeats safely.

---

Section 5: The saga, or giving up atomicity on purpose

If agreement can't be guaranteed in bounded time and 2PC stalls when its coordinator fails, most microservice systems settle for something weaker: the saga. A saga splits one big transaction into a sequence of local transactions, each fully ACID inside its own service's database, chained so that each step triggers the next. If a step fails, the saga runs compensating transactions that undo the steps that already committed, in reverse order.

The idea is older than microservices. Hector Garcia-Molina and Kenneth Salem introduced sagas in 1987 for long-lived transactions inside a single database, the kind that would otherwise hold locks for hours. Their paper already makes the point that matters most: a compensation undoes an action "from a semantic point of view," meaning it reverses the business effect, without necessarily restoring the old data. Their example is an airline booking, where the compensation cancels the seat instead of resetting the seat count, because other bookings may have changed the count in the meantime.

Here's checkout as a saga, ordered with care:

1. Reserve the item in inventory, as a hold that expires on its own after, say, 15 minutes. Compensation: release the hold.
2. Authorize the card, which places a hold on the customer's funds without moving any money. Compensation: cancel the authorization, which releases the hold and costs nothing.
3. Capture the payment, which actually moves the money. This is the point of no return.
4. Confirm the reservation, create the shipping label, and send the confirmation email. These can't be undone in any useful way, so they're built to succeed eventually: retried until they work, with a person alerted if they keep failing.

Figure 9. The checkout saga, laid out by kind of step. Solid arrows are the forward path; dotted arrows are what happens on failure. Before the pivot, failure runs compensations in reverse order (cancel the authorization, then release the hold). After the pivot, failure never undoes anything; it retries, and eventually asks a person. That's why every irreversible action lives to the right of the pivot.

Steps 1 and 2 follow a pattern sometimes called Try-Confirm/Cancel (TCC): each service first places a tentative hold, then later confirms or cancels it. Chris Richardson's Microservices Patterns gives names to the three kinds of step: compensatable transactions that can be undone (steps 1 and 2), a single pivot transaction that is the go or no-go point (step 3), and retriable transactions after the pivot that must never fail permanently (step 4). Every step before the pivot needs a compensation. Every step after it needs a retry policy instead.

This ordering also keeps compensations cheap. Cancelling an authorization in step 2 means no money ever moved. Compare refunding a captured payment: with Stripe, for example, the processing fees on the original charge aren't returned when you refund. Authorizations also don't last forever, which gives the saga a deadline: Stripe's documentation lists seven days for customer-initiated online card payments, and under five days for Visa merchant-initiated ones (as of 2026), after which the hold is released if you haven't captured.

There are two ways to run the chain. In choreography, there's no central controller: each service publishes an event when its step completes ("payment authorized"), and the next service listens for it and acts. It's quick to start and hard to debug, because the workflow exists only as the sum of everyone's subscriptions, and "where is order 4821 stuck?" means searching five services' logs. In orchestration, a dedicated orchestrator sends each command, records which step each order has reached, and triggers compensations when something fails. AWS Step Functions is a managed orchestrator, durable-execution engines such as Temporal, which record every step of a workflow so it can resume exactly where it stopped after a crash, are another option, and plenty of teams build a small state machine on top of their own database. It's one more component to run, but the workflow is written down in one place. My own rule: two or three steps can be choreographed, and past that I want an orchestrator, because nobody can hold a ten-step choreography in their head during an incident.

Figure 10. Two ways to run the same saga. In choreography (top), the workflow exists only as the chain of who listens to whom, so answering "where is this order?" means reading several services. In orchestration (bottom), one component sends every command and records every reply, so the workflow and its current state live in one place.

Figure 11. The orchestrator's state machine for one order. Every order is in exactly one state, stored in a table, so a crashed orchestrator resumes from the stored state. Notice there is no arrow from Fulfilling to Compensating: once past the pivot, the only ways out are success or a person.

In code, the orchestrator's memory is a table like this one. Each row is one order's saga, and the correlation id is the thread that ties every command, event, log line, and trace for that order together (Section 7 explains why that matters).

CREATE TABLE checkout_saga (
  order_id        bigint PRIMARY KEY,
  state           text        NOT NULL,  -- RESERVING, AUTHORIZING, CAPTURING, FULFILLING,
                                         -- COMPENSATING, DONE, CANCELLED, NEEDS_ATTENTION
  step_attempts   int         NOT NULL DEFAULT 0,
  deadline        timestamptz NOT NULL,  -- e.g. when the card authorization expires
  correlation_id  uuid        NOT NULL,  -- carried on every message, log line, and trace
  last_error      text,
  updated_at      timestamptz NOT NULL DEFAULT now()
);

-- Advancing a step is a guarded update, so two orchestrator instances
-- can't both move the same order forward.
UPDATE checkout_saga
   SET state = 'CAPTURING', step_attempts = 0, updated_at = now()
 WHERE order_id = 4821 AND state = 'AUTHORIZING';
-- 0 rows updated means someone else already advanced it: stop.

Compensations also have to tolerate arriving out of order. Suppose the inventory call timed out, so the saga gives up and sends "release hold", but the original reserve request was only slow and lands afterward. The release must succeed even when it finds nothing to release, and it should leave a "cancelled" marker for that order so the late reserve is rejected rather than creating an orphaned hold.

Figure 12. A compensation that overtakes the step it undoes. If step 3 had failed because there was nothing to release, or hadn't left a marker, the late reserve in step 4 would have created a hold that no saga would ever release. Compensations must succeed on "not found" and must leave a record that blocks the original action from landing late.

Some actions can't be compensated at all: the email already sent, the partner API already called, the loyalty points already spent, the box already on the truck. That's why they come after the pivot. And when even a retriable step keeps failing, the saga ends in a "needs attention" state, with full context for a person to act on, rather than sitting silently half-finished.

---

Section 6: The outbox, or how events actually leave the building

Sagas run on events, and publishing an event has a small version of the same problem. The payment service commits "authorized $50" to its database, then publishes "payment authorized" to the message broker, the system (Kafka or RabbitMQ, for example) that carries events between services. If the process crashes between the commit and the publish, the money is held and nobody downstream ever hears about it. If it publishes first and then the commit fails, inventory acts on a payment that never happened. Writing to two systems with no shared transaction is called the dual-write problem, and it's Section 2's ambiguity squeezed into a single service.

The transactional outbox fixes it by never writing to two systems at once. The service writes the event into an outbox table in the same local transaction as the business change. One commit covers both, so the authorization and its event are saved together or not at all. A separate relay then reads the outbox and publishes each event to the broker. There are two common relays. A polling publisher queries the table for unsent rows. Change data capture (CDC) instead tails the database's transaction log, the same log replicas read, so the relay adds almost no load to the database. Debezium is the best-known CDC tool, and it ships an "outbox event router" built for exactly this pattern.

Figure 13. Outbox on the way out, inbox on the way in. Neither service ever writes to two systems in one step: the payment service writes only to its own database, the relay moves events from that database's log to the broker, and the inventory service records the event id and its effect in one local transaction. Delivery is at least once; because of the inbox, the effect happens once.

The details that make it production-grade:

• Ordering. Kafka splits each topic into partitions, ordered logs that are processed in parallel, and picks a message's partition from its key. Debezium's router uses the outbox row's aggregate id (an aggregate is a cluster of data treated as one unit, like an order with its line items; the aggregate id is that order's id) as that key, so all events for one order land on the same partition and stay in order. Events for different orders have no ordering guarantee between them.
• Duplicates. If the relay crashes after publishing but before recording its progress, it publishes some events again. Each event carries a unique id, and consumers use it to skip repeats.
• Table growth. With log-based CDC, a common trick is to insert the outbox row and delete it in the same transaction. The insert still appears in the transaction log, so it still gets published, but the table stays empty. Gunnar Morling's 2019 Debezium post describes this.
• The consumer's side. The mirror image of the outbox is an inbox: the consumer records each processed event id in the same local transaction as the change that event causes. A duplicate then finds its id already recorded and does nothing. An outbox on the sending side plus an inbox on the receiving side gives you an exactly-once effect over at-least-once delivery, which Section 4 says is the most any system can offer. One limit: the inbox only protects changes inside the consumer's own database. If handling the event calls an outside service, like a payment provider, pass an idempotency key derived from the event id so that service can deduplicate too.

Here is the pattern in PostgreSQL. The comments mark the details that are easy to get wrong.

-- The outbox lives in the same database as the business tables.
CREATE TABLE outbox (
  id              uuid        PRIMARY KEY,   -- event id: consumers deduplicate on it
  aggregate_type  text        NOT NULL,      -- e.g. 'payment'
  aggregate_id    text        NOT NULL,      -- e.g. the order id; becomes the message key
  type            text        NOT NULL,      -- e.g. 'PaymentAuthorized'
  payload         jsonb       NOT NULL,
  created_at      timestamptz NOT NULL DEFAULT now(),
  published_at    timestamptz               -- used only by a polling relay
);

-- The business change and its event commit together, or not at all.
BEGIN;
  UPDATE payments SET status = 'authorized' WHERE order_id = 4821;
  INSERT INTO outbox (id, aggregate_type, aggregate_id, type, payload)
  VALUES (gen_random_uuid(), 'payment', '4821', 'PaymentAuthorized',
          '{"order_id": 4821, "amount_cents": 5000}');
COMMIT;

-- A polling relay. SKIP LOCKED lets several relays share the work, but then
-- events for one order can be published out of order across relays; if order
-- matters, run one relay per partition of aggregate ids, or use log-based CDC.
-- Even one relay can misorder: created_at is the transaction's start time, not
-- its commit time, so rows can become visible out of created_at order.
-- Log-based CDC follows commit order.
BEGIN;
  SELECT id, aggregate_id, type, payload
    FROM outbox
   WHERE published_at IS NULL
   ORDER BY created_at
   LIMIT 100
   FOR UPDATE SKIP LOCKED;
  -- publish each row to the broker and wait for its acknowledgment, then:
  UPDATE outbox SET published_at = now() WHERE id = ANY(:published_ids);
COMMIT;

-- The consumer's inbox: the duplicate check and the effect share one transaction.
CREATE TABLE inbox (event_id uuid PRIMARY KEY, processed_at timestamptz DEFAULT now());

BEGIN;
  INSERT INTO inbox (event_id) VALUES (:event_id)
  ON CONFLICT (event_id) DO NOTHING;
  -- if that inserted 0 rows, this event was already handled: ROLLBACK and stop
  UPDATE stock SET held = held + 1 WHERE sku = 'A-100';
COMMIT;

There's one more way to get events out reliably, and it's worth knowing because teams often reach for it: event sourcing. Martin Fowler's 2005 description is that it "ensures that all changes to application state are stored as a sequence of events." Instead of updating a payments row and also recording an event, the service only appends events ("PaymentAuthorized", "PaymentCaptured"), and the current state of a payment is computed by replaying them, usually with periodic snapshots so a replay doesn't start from the beginning of time. The dual-write problem disappears as long as other services read the event store itself, or a relay that tails it (publishing to a separate broker from application code would bring the problem back), and the event log doubles as a full audit history, which payment and ledger systems value. The costs are real, though. Every query about current state needs a projection (a read model built from the events and kept up to date), and because events can never be changed once written, evolving their schema takes ongoing care. My view: event sourcing is worth it when the history itself is the product, like a ledger, and an outbox is the better default when you just need reliable events next to an ordinary database.

Figure 14. Two ways to make "the change" and "the event" the same thing. With an outbox (top), the row is the truth and the event is a reliable side effect. With event sourcing (bottom), the event is the truth and every view of current state, including the one the service itself queries, is derived from the log.

What's left are operational failures, and none of them lose data. The relay can fall behind, so you graph its lag like any consumer lag and page on it. The broker can be down, in which case events wait in the outbox, which is just a table. A malformed event can wedge the relay, so after a few retries it goes to a dead-letter queue, a side queue for messages that keep failing, with a named owner. Each of these delays events visibly, which is the failure mode you want from a system that chose repair over certainty.

---

Section 7: Where each approach bites

Knowing the patterns is the entry-level answer. Knowing how each one fails is the next level, because they all fail, each in its own particular way.

2PC bites at the coordinator. It works until the worst possible moment: a coordinator failure between the phases leaves every prepared participant holding locks. Your atomicity is only as available as your coordinator's recovery. That's why plain 2PC lives inside systems where the coordinator is replicated or comes back in seconds, and almost never across independently deployed services. If someone proposes it across services, ask "how many minutes of frozen locks does a coordinator crash cost us, and who restarts it?" That question usually ends the proposal.

Sagas bite through missing isolation. Between "authorized" and "captured," the system is in a state no single-database transaction would ever expose, and other readers can see it. A revenue dashboard counts money that might be released. A second saga can read stock that's about to be held and oversell it. These are isolation anomalies, such as lost updates, where one write silently overwrites another, and dirty reads, where someone acts on data that later gets undone. Richardson calls the design fixes countermeasures, and these four are the most useful:

• Semantic lock: a "pending" flag on the record, set by an early step and cleared by a later one, that other workflows check and respect.
• Commutative updates: design changes so their order doesn't matter, like "add 1" and "subtract 1" instead of "set to 5."
• Reread value: confirm the record hasn't changed before overwriting it. This is optimistic locking: no lock is held while you work, and the write succeeds only if a version number still matches what you read.
• Pessimistic view: order the steps so the riskiest changes become visible last.

Figure 15. An isolation anomaly between two sagas. Each saga's steps are individually correct transactions; the bug is that the read and the hold are separate steps, so both sagas act on the same stale read. Both fixes move the check and the change into one atomic statement or make the in-between state visible to everyone.

A common failure is a saga designed as if its steps were instant. In testing, step 1 and step 2 are milliseconds apart. Under load they're minutes apart, and every anomaly window grows with the latency.

Sagas also bite through missing compensations. The team that writes ten forward steps and two compensations has a saga that unwinds partway and stops. Review it step by step: what undoes this, what's its retry policy, and what happens if the compensation itself fails? "The email can't be unsent" is a fine answer when it was a deliberate choice and the email comes after the pivot.

A small, real example of how compensation coverage goes wrong comes from GitHub's engineering blog in September 2009, in a post titled "Revenge of the double billing." A billing batch ran twice. The team ran a void batch to cancel the duplicate charges, but, in the post's words, "users whose card had declined in a previous batch and were being retried did not have the duplicate charge voided." The compensation handled the common case and missed one state, and the remaining customers had to be found and refunded by hand. Two lessons carry over to any saga: enumerate every state a compensation can meet, including retries in progress, and make the forward step idempotent (a billing run keyed by invoice and period can't charge twice), so the compensation is needed less often.

The outbox bites through lag and ordering. Events arrive at least once, so every consumer has to be idempotent, or a repeated "payment authorized" reserves stock twice. Ordering holds per key, not across keys, so a consumer that needs events from two different orders in sequence has to cope with them arriving the other way round. And the relay is new infrastructure: its lag is a first-class metric, and its dead-letter queue needs an owner.

Sagas bite in production when you can't see them. A saga spreads one business operation across several services and several minutes, so the tools that work for one request in one service stop being enough. Four habits close the gap:

• Carry a correlation id everywhere. Generate one id per saga, put it on every command, event, log line, and trace span, and "where is order 4821 stuck?" becomes one search instead of an investigation.
• Trace across the broker. Distributed tracing records each unit of work as a span, with its timing, and links spans into one trace using a trace context, a few ids passed along with each request. OpenTelemetry is the common standard, and it can link spans across asynchronous hops if the trace context is copied into message headers. Then a slow saga shows up as one timeline with the slow step highlighted.
• Test the failures on purpose. Inject a crash after every step and between every commit and publish, run the saga, and assert that it ends in Done, Cancelled, or Needs attention, never stuck in between.
• Version your events. Events outlive the code that wrote them, especially with retries and replays. Use a schema format with compatibility rules, for example Avro or Protobuf (compact binary formats with explicit schemas) plus a schema registry (a service that stores every schema version and rejects incompatible changes), only add optional fields, and make consumers ignore what they don't understand.

Figure 16. Making a saga observable. The correlation id and trace context ride along on every command and every event, including through the outbox and the broker, so the logs and traces of all four services join into one story per order.

The deepest bite is conceptual. Sagas don't give you atomicity or isolation. They give you a managed, observable path through partial states toward consistency, with compensations for the failures you expected and a person for the rest. Teams that adopt sagas expecting ACID get distributed complexity without the guarantees they thought they were buying. Teams that adopt them expecting managed inconsistency get a system that fails in ways they can see and repair.

---

Section 8: The numbers, and a decision framework

Rough numbers sort designs into buckets. Treat these as orders of magnitude, because your hardware and network will move them:

Operation	Typical cost	What dominates it
Local commit	about a millisecond, up to a few	the disk flush, plus a replica's acknowledgment if replication is synchronous
2PC commit	several times a local commit; MySQL distributed transactions reported at over 10x single-node (via DDIA)	two round trips to every participant, durable log writes, locks held throughout
2PC after a coordinator crash	as long as the coordinator takes to recover, often minutes	in-doubt participants holding locks
Saga, end to end	the sum of its steps plus a broker hop per step: milliseconds when healthy, minutes under backlog	the slowest step and the queue depth
Saga p99	estimate with the sum of each step's p99 plus broker hops, a pessimistic figure; queue waits between steps often dominate the tail	p99 is the latency 99% of requests come in under (#9)
Outbox relay lag	under a second normally	broker and database health; this is the number you graph and page on
Card authorization lifetime	7 days for customer-initiated online card payments; under 5 for Visa merchant-initiated (Stripe, 2026)	the hard deadline for a saga that authorizes before it captures
Partial outcomes	2^n − 2 for n participants: 2, 6, 30 for 2, 3, 5	how many services one business operation touches

The decision framework, in the order I'd ask the questions:

1. Can you avoid the distributed transaction? Redraw the boundaries so one service owns the whole unit of work. A payment service that also owns the ledger entries for each charge needs no distributed anything. This is the most underused option, and Section 9 makes the case for it.
2. How bad is temporary inconsistency? If the business can live with "authorized, not yet captured" for seconds or minutes, use a saga. If it truly can't tolerate any visible intermediate state, you need real atomicity, and you should push back on that requirement first, because it often softens under questioning.
3. If you need real atomicity, where will it come from? Get it from a database that replicates its coordinator (Spanner and the distributed SQL databases built in its image) rather than running XA across services you operate separately.
4. Can every step before the pivot be undone? If each has a sensible compensation (release, cancel, refund), a saga fits. If an irreversible step would have to happen early, reorder until it doesn't, or go back to question 1.
5. Who operates it? Sagas need compensation coverage, idempotent consumers, relay-lag monitoring, and a dead-letter owner. 2PC needs recovery runbooks for the coordinator. Count the operational surface before choosing, because a pattern nobody on call understands is the wrong pattern.

Figure 17. Section 8's questions as a flowchart. Start at the top and stop at the first box that answers your situation. Every path that ends in a saga goes through the same final box, because the operational work isn't optional for any of them.

---

Section 9: The principal take: design the boundary, not the protocol

The staff- and principal-level answer starts with a refusal: the best distributed transaction is one you designed away. Every protocol in this post manages the ambiguity from Section 2, and the cheapest way to manage ambiguity is to shrink the places where it can appear.

Pat Helland made this argument in "Life Beyond Distributed Transactions," first at CIDR in 2007 and in a revised version for ACM Queue in 2016. His claim is that systems built to scale should treat a single entity, one order or one account, as the boundary of atomicity; should never span an atomic transaction across entities; and should assume every message between entities can arrive more than once, so every handler must be idempotent. Nearly twenty years later, it still reads like a description of how large systems actually work.

The practical moves, in order of preference:

1. One owner per unit of work. Give a single service ownership of each all-or-nothing unit, so its transaction stays local and fully ACID. The order service owns the order's lifecycle. The payment service exposes "authorize" and "capture" as operations, but the order service decides when to call them.
2. Share facts instead of coordinating. Rather than service A asking service B to agree, A publishes what happened ("payment captured") and B updates its own copy of the data it needs. Martin Fowler calls this event-carried state transfer. B never needs A's agreement, only A's events, delivered reliably by the outbox from Section 6.
3. Orchestrated sagas for workflows that truly cross owners and need undo semantics, with the pivot placed on purpose.
4. 2PC inside a database built for it when cross-entity atomicity is worth the cost, with the coordinator replicated by consensus. Hand-rolled XA across services stays off the list.

Figure 18. The principal move, drawn. On the left, one business operation needs three databases to agree. On the right, the boundary has been redrawn so the all-or-nothing part (a payment and its ledger entries) lives in one database, and the other service learns about it through an event instead of a vote.

Underneath all four is one habit: know which inconsistency you're tolerating, for how long, and who can see it. "Eventually consistent" is where that conversation starts, and it needs three follow-up answers. Eventually when: seconds or minutes, and what's the SLO (the reliability target the team commits to)? Consistent for whom: which readers can see the intermediate state, and does any of them act on it? And what repairs it: automatic reconciliation, a compensation, or a person with a runbook? A design that answers those three has a consistency story; one that stops at "eventually consistent" is still waiting for one.

Atomicity is a property of your boundaries, not your protocols. Draw the boundaries so each all-or-nothing unit fits inside one database, and most of the protocols become unnecessary. Where a unit truly can't fit, use an orchestrated saga with an outbox, idempotent consumers, measured relay lag, and compensations that someone actually wrote and tested.

---

All or nothing, distilled

Key numbers:

Metric	Value
Bad partial outcomes with n participants	2^n − 2 (2 services: 2; 3: 6; 5: 30)
2PC network cost	at least two round trips to every participant, plus durable log writes
2PC slowdown, reported	MySQL distributed transactions over 10x slower than single-node (via DDIA)
2PC blocking window	the coordinator's recovery time, often minutes, with locks held
Card authorization validity	7 days customer-initiated online; under 5 for Visa merchant-initiated (Stripe, 2026)
Cancelling an authorization vs refunding	cancel moves no money; Stripe doesn't return the original processing fees on refunds
Outbox relay lag	under a second normally; graph it and page on it
Saga p99	estimate as the sum of step p99s plus broker hops (pessimistic); queue waits dominate the tail

Every trade-off, in one table:

Decision	Chose	Over	Why
Cross-service atomicity	Saga	2PC/XA across services	no coordinator crash can freeze everyone; you pay in lost isolation
Running the saga	Orchestration past two or three steps	Choreography	the workflow lives in one readable, debuggable place
Step order	Compensatable, then pivot, then retriable	Whatever order the code grew in	steps that can't be undone never need undoing
Payment step	Authorize, then capture	Charge immediately	cancelling an authorization is free; refunds cost fees
Publishing events	Transactional outbox	Write the database, then publish	a crash between the two loses or invents events
Outbox relay	Log-based CDC	Polling the table	almost no extra database load, events in commit order
Consumer safety	Inbox and idempotent handlers	Trusting exactly-once delivery	delivery is at least once; the effect can still be once
When atomicity is required	A database with a replicated coordinator	Hand-rolled XA	a coordinator crash becomes a failover, not a freeze
Events next to an ordinary database	Outbox	Event sourcing	event sourcing pays off when history is the product, like a ledger
Late or repeated messages	Compensations that accept "not found" and leave markers	Assuming order	a slow forward step can land after its own undo
Production visibility	Correlation id + traces through the broker	Per-service logs only	one query answers "where is this order?"
Boundaries	One owner per unit of work	Any coordination protocol	the protocol you don't need is the cheapest one

Three ideas to take with you:

1. The ambiguity is permanent. You can't tell "failed" from "slow" across a network. Choose what happens inside that window: wait for certainty (2PC) or act and repair (sagas plus reconciliation).
2. Order the saga around its pivot. Undoable steps first, then the single go or no-go step, then steps that only ever need retries. Authorize before you capture.
3. Atomicity is a boundary decision. Redraw boundaries first so all-or-nothing units fit in one database, and reach for protocols only for what truly can't fit.

Further reading

• Sagas. Hector Garcia-Molina and Kenneth Salem, SIGMOD 1987. The original paper, short and readable, including the airline example.
• Life Beyond Distributed Transactions: An Apostate's Opinion. Pat Helland, ACM Queue 2016 (first published at CIDR 2007). The case for entities as the boundary of atomicity.
• Consensus on Transaction Commit. Jim Gray and Leslie Lamport. Paxos Commit, with classic 2PC as its single-coordinator special case.
• Impossibility of Distributed Consensus with One Faulty Process. Fischer, Lynch, and Paterson, Journal of the ACM, 1985. The FLP result from Section 4.
• Pattern: Saga. Chris Richardson. Choreography versus orchestration and the missing-isolation problem; his book Microservices Patterns covers pivot transactions and countermeasures in depth.
• Reliable Microservices Data Exchange With the Outbox Pattern. Gunnar Morling, 2019. The outbox with log-based CDC, including the insert-then-delete trick and consumer deduplication.
• Outbox Event Router. Debezium docs: the outbox table layout, aggregate-id keys for ordering, and event ids for deduplication.
• PREPARE TRANSACTION. PostgreSQL docs, including the warning to keep prepared transactions disabled unless a transaction manager is closing them out.
• Event Sourcing. Martin Fowler, 2005. The original description of events as the source of truth, including rebuilds and replays.
• Revenge of the double billing. GitHub, 2009. A short, candid account of a compensation that missed one state.
• Designing robust and predictable APIs with idempotency. Brandur Leach, Stripe, 2017. Idempotency keys as the practical tool for safe retries.
• Place a hold on a payment method. Stripe docs on authorizing now, capturing later, and how long authorizations last.
• Designing Data-Intensive Applications. Martin Kleppmann. In the first edition, Chapter 7 covers transactions and isolation, and Chapter 9 covers 2PC, consensus, and the limits of both.

Where you'll meet this

Checkout and order systems are the obvious place, but the same problem shows up wherever one business action touches more than one service: moving money between ledgers, booking a flight plus a hotel plus a car (the saga paper's own home ground), provisioning cloud resources across services, and any service that writes to its database and publishes an event. Step 5 of the Twitter/X walkthrough (Design Walkthrough #2) hits exactly that: saving a tweet and publishing its fan-out event is a dual write, and the outbox is the fix. In interviews, "design a payment system," "design a hotel booking system," and "design a ticketing system" all turn on this post's material.

Next up: the single-database guarantees from Section 1, isolation levels included, get the full treatment in The Database Menu: SQL vs NoSQL (#16).

Keep exploring

• Anchor: From Laptop to Billions of Clicks: Designing a URL Shortener in 11 Steps
• Design Walkthrough #2: From One Timeline to 500 Million Tweets a Day: Designing Twitter/X in 11 Steps
• #1 The 100:1 Superpower: Caching
• #2 The Blast Radius: Surviving the Day Your Dependencies Fail
• #3 Divide and Conquer: Sharding and Partitioning
• #4 The Snowflake Problem: Unique IDs at Scale
• #5 Copies of the Truth: Replication
• #6 Do No Harm Twice: Idempotency
• #7 The Copy at the Doorstep: CDNs and Edge Computing
• #8 The Bouncer's Math: Rate Limiting
• #9 What Broke at 3 AM: Observability, p99, and Useful Alerts
• #10 Do It Later, On Purpose: Async Processing and Queues
• #11 The Traffic Cop: Load Balancing
• #12 Assume They're Already Knocking: Security and Abuse at Scale
• #13 Do the Math First: Estimation for System Design
• #14 The Whiteboard Playbook: Taking On Any System Design Challenge
• #15 All or Nothing: Distributed Transactions (this post)
• #16 The Database Menu: SQL vs NoSQL

---

Blueprints of Scale — Core Concepts #15. If this helped, the best thanks is a share with someone who's learning.