Same Data, Different Answers: Consistency, Explained Like You're New

You change your display name. The page says saved. You refresh, and the old name is back. You refresh again, and the new name returns. Nothing crashed. No one else edited your profile. The database simply had more than one copy of your name, and your two refreshes asked two different copies.

That small, unsettling moment is the whole subject of this post. Once a system keeps copies, and every system at scale keeps copies, the copies will sometimes disagree for a short time. Consistency is the name for the promise you make about that disagreement: how stale an answer is allowed to be, how out of order answers are allowed to look, and who is allowed to see a version that is not final yet.

If you're new to this, Sections 1 and 2 need no background. Sections 3 through 7 are the working machinery: the spectrum of guarantees, the difference between linearizability and serializability, quorum math, the CAP theorem stated honestly, and the anomalies that appear when guarantees get weak. Sections 8 and 9 are the part you will use at work: choosing a guarantee per operation, and making a weak guarantee safe in the product. Section 10 is the principal-level take: bounded staleness, testing the promise, and what the products actually let you buy. There's a one-page summary at the end under Same data, different answers, distilled, and a Further reading list of professional engineering sources that go deeper.

This post builds on three earlier ones, and it does not repeat them. Replication (#5) covered how copies are made and how failover works. Distributed Transactions (#15) covered all-or-nothing across services. SQL vs NoSQL (#16) used CAP in one line to help pick a database. This post is the missing middle: once you have copies and you have picked a database, what exactly did you promise your users about the answers those copies give?

Section 1: The update that went backwards

Start with the profile name, because it is the version everyone has lived.

A service stores each profile on three machines, for durability and for read scale. You submit the new name. One machine, the leader for your profile, records it and replies saved. The other two copies get the update a moment later, over the network. In that moment, usually well under a second, the system contains both names at once. A refresh that lands on the leader shows the new name. A refresh that lands on a copy that has not caught up shows the old one. A third refresh can go either way. From your chair, your own update went backwards.

Nothing here is a bug in the ordinary sense. Every machine did what it was told. The gap is the design: the system accepted your write before every copy agreed, because waiting for every copy on every write is slow, and sometimes impossible when a copy is down. So it traded a short window of disagreement for speed and availability. The trade itself is normal. What separates a good system from a confusing one is whether that trade was chosen on purpose, for this kind of data, with a limit on how long the disagreement can last.

Three plain questions organize the entire post:

How fresh? After a write succeeds, how soon must every later read see it: immediately, after a bounded delay, or eventually with no bound promised?
How orderly? If two writes happen in a known order, must every reader see them in that order? What about writes that caused other writes?
Whose view counts? Is the promise the same for the writer reading their own change, for another user, and for a background job an hour later?

Different answers to those three questions are what engineers mean by consistency models. The models have intimidating names, but each one is just a bundle of answers to those questions.

Section 2: What consistency actually promises

A common beginner confusion, worth clearing early: consistency in this post is not about whether the data is correct. A system can be perfectly consistent and store a wrong name, a wrong price, or a wrong balance, as long as every reader sees the same wrong value in the promised order. Correctness against business rules is a separate concern, handled by validation, constraints, and transactions (#15). Consistency here is about visibility and ordering: when a write becomes visible, and in what order writes become visible to readers.

One more clarification, because the same word does double duty. The C in ACID, the single-database guarantee from #16, is about business rules holding across a transaction. The C in CAP, covered in Section 6, is closer to what this post means: every read receives the most recent write, or an error. Same letter, related spirit, different promise. When someone says consistency in a design discussion, ask which one they mean. Half of all consistency arguments are two people using the word differently.

It also helps to think of a consistency model as a contract written in terms of forbidden surprises. A strong contract forbids many surprises: no stale reads, no going backwards, no seeing effects before their causes. A weak contract forbids few: it promises only that, if writes stop, the copies will converge to the same value at some point. Everything in Section 3 is a contract somewhere between those two, and each step down the ladder drops a forbidden surprise in exchange for speed, availability, or lower cost.

Why would anyone drop a protection on purpose? Because forbidding surprises requires coordination, and coordination requires waiting. To guarantee that a read sees the latest write, the system must either ask the machine that knows the latest write, or make writers wait until enough copies agree before calling the write done. Both cost time. Across a room, that time is a millisecond or two. Across an ocean, it is 130 to 180 milliseconds for a round trip between, say, the US East Coast and East Asia, and you pay it on the critical path, where a user is watching a spinner. Section 5 does this arithmetic in full. For now, hold the shape: stronger promise, more waiting; weaker promise, less waiting and more surprises to design around.

Section 3: The spectrum, in plain language

Here is the ladder, weakest to strongest. Read it as a set of promises you can buy, each one costing more coordination than the one before.

Eventual consistency: if no new writes arrive, all copies will at some point hold the same value. That is the whole promise. A read can return any older value in the meantime, two readers can see different values at the same moment, and a reader can see a value go backwards, as in Section 1. DNS is the everyday example: change a record, and the world converges over minutes to hours. Product catalogs, view counts, and like counts live happily here.

Read your writes: you, the writer, will always see your own completed writes. Other people might still see an older value for a while. This one small upgrade fixes the most jarring version of the Section 1 story: your own update never appears to go backwards to you. Most users judge a system by their own view, so this guarantee buys a lot of perceived correctness for a modest cost. Profile edits, settings changes, and posts in a feed are natural fits.

Monotonic reads: once you have seen a value, you will never later see an older one. Your view can lag behind the newest write, but it only moves forward. Without it, a user can watch a comment thread lose replies between refreshes, which reads as data loss even when nothing was lost.

Writes follow reads: if you read a value and then write because of what you read, your write is ordered after the version you read, for anyone who later sees it. You reply to the corrected post, and nobody sees your reply attached to the old, wrong version as if you had answered that. It sounds obvious. Eventual systems do not owe it to you.

Monotonic writes: your own writes take effect everywhere in the order you made them. Edit a name twice in a row, and no copy applies the second edit before the first. Without it, copies can hold states that correspond to no order you ever produced.

These four, read your writes, monotonic reads, writes follow reads, and monotonic writes, have a collective name: the session guarantees. They are the bundle product teams actually buy together, and it is worth seeing why. Each one only constrains the view of a single client in a single session: what you see after you act, and how your own actions line up. Because the promise is scoped to one session rather than to all readers at all times, it is cheap to keep: a version token carried with the session, or a rule that a session only reads from copies that have caught up to its last write, enforces all four without any global coordination. And jointly they remove nearly every complaint that makes users say an app feels broken: my edit vanished, my reply landed on the wrong post, my change undid itself. Section 10 shows a major product selling exactly this bundle under the name session.

Session guarantee flow: a write returns a version token and the next read waits for a replica that has caught up to that token

Causal consistency: if one write caused another, nobody sees the second without the first. If a post exists because it was written, and a comment exists because someone read the post, no reader sees the comment floating above a missing post. Writes that are unrelated can appear in different orders to different readers. Causal consistency is often the sweet spot for social and collaborative features: it forbids the confusing surprises, and allows the harmless ones.

Sequential consistency: every reader sees writes in the same single order, and that order respects each writer's own order, but it does not have to respect real time. A write that finished before another began can still appear after it in the shared order, as long as everyone sees the same order. Stronger than causal, weaker than the next rung, and mostly useful as a stepping stone in understanding.

Linearizability, usually sold as strong consistency: every operation appears to take effect at a single instant between when it started and when it finished, and later operations see earlier ones. If a write completed before a read began, in real time, that read must see the write or a newer one. To users, the system behaves as if there were only one copy. This is the strongest promise on the ladder for a single item, and the most expensive to keep across regions and during failures.

Timeline comparing a linearizable read, which must return the new value, with an eventual read that can return a stale value

Two honest notes about the ladder. First, real products rarely pick one rung for everything. The same app can be linearizable for a password change, read-your-writes for a profile edit, causal for comments, and eventual for a like count. Per-operation choice is normal engineering, not inconsistency about consistency. Second, vendor labels do not map cleanly onto the ladder. One product's strong can be another product's marketing. The test is always the three questions from Section 1: how fresh, how orderly, whose view.

Guarantee	The promise, in one line	A surprise it still allows
Eventual	Copies converge if writes stop	Stale reads, values moving backwards
Read your writes	You always see your own completed writes	Others see an older value for a while
Monotonic reads	Your view never moves backwards	Your view lags the newest write
Writes follow reads	Your writes build on the versions you read	Other readers may not see the chain yet
Monotonic writes	Your writes apply in the order you issued them	Two different writers still interleave freely
Causal	Effects never appear before their causes	Unrelated writes appear in different orders
Sequential	Everyone sees one shared order	That order can differ from real time
Linearizable	Behaves as if there were one copy	None on freshness or order, at a price

Section 4: Linearizable is not serializable: the two strong words

The top of the ladder hides a confusion that senior engineers carry for years, because the two strongest promises in this field sound like synonyms and are not. Linearizability and serializability answer different questions, about different units of work, and a system can honestly offer one without the other.

Linearizability is about one item. It says a single key, register, or object behaves as if there were exactly one copy of it, and every operation takes effect at one instant between its start and its finish. Its extra ingredient is real time: if a write finishes before a read begins, in the actual world, that read must see the write or something newer. Once operation A completes before operation B starts, B is not allowed to act as if A had not happened yet. This is the guarantee Section 3 put at the top of the ladder, and its home ground is single items: a lock, a leader lease, a configuration flag, one user's session record.

Serializability is about transactions across many items. It says the result of running a set of transactions concurrently is the same as if they had run one after another, in some serial order. That is the whole promise. It says nothing about which order, and in particular it does not require the serial order to match real time. A transaction that started after another one finished may still be placed before it in the equivalent serial order, as long as the final state and every returned value are consistent with that placement.

Here is the gap, made concrete. A transaction moves a price from 9 to 10, writing two items, a catalog entry and a search index entry, and it completes at 10:00:05. A second transaction starts at 10:00:06, after the first is fully done, and reads both items. It sees 9 and 9, the old price twice. Is that history serializable? Yes: it is exactly what would have happened if the reading transaction had run before the writing one, so a valid serial order exists. Does it keep the single-copy illusion? No: the first transaction finished in real time before the second began, so any system promising real-time order would have to show 10. A database can truthfully advertise serializable isolation and return those 9s all day. Serializable, and not linearizable, in one small story.

A history that is serializable but not linearizable: a later transaction reads the old price twice

The combination has its own name: strict serializability, serializability plus the condition that the serial order respects real time for transactions that do not overlap. Google Spanner is the famous example, and it uses the term external consistency for the same idea: if a transaction commits before another begins in the real world, the second is ordered after the first, worldwide. That is the strongest promise a multi-item system commonly sells, and Section 10 notes what it costs to keep: specialized clock infrastructure, and commit waiting, deliberately sitting still so that real-time order stays true.

Which systems promise which? Single-item stores and coordination services sell linearizability per item: etcd and ZooKeeper operations on a key, a strongly consistent read of one DynamoDB item. They mostly do not offer multi-item transactions at all, so serializability never comes up. Relational databases sell the other word: serializable isolation in PostgreSQL means your transactions are equivalent to some serial order, with no standing promise that the order matches the wall clock across machines, replicas, or read-only followers. Plenty of design reviews go wrong right here, when a team hears serializable, assumes the single-copy illusion from Section 3, and builds something that needed linearizability, such as treating a shared row as a lock or as the one queue that hands out work.

The working rules are short. For one item whose latest value decides something, a lock, a lease, a flag, ask whether reads are linearizable. For an invariant spanning several items, the on-call rule in Section 7, a transfer between two accounts, ask whether transactions are serializable. If people outside the system act on the order they observed, so the order itself becomes a fact in the real world, ask for strict serializability, and expect to pay Spanner prices for it.

Section 5: Why strong costs, and the quorum shortcut

Strong promises cost coordination, and coordination has a price list.

The clearest way to see it is the quorum, the voting pattern introduced for elections in #5 and used here for reads and writes. Take three copies, call the count N = 3. A write is done when W copies have confirmed it. A read consults R copies and takes the newest answer among them. The arithmetic that matters is one line:

If W + R is greater than N, the read set and the write set must overlap.

With N = 3, W = 2, and R = 2, the sum is 4, which is greater than 3, so any read of two copies must include at least one copy that took the latest acknowledged write. That overlap is how a system can offer a strong-looking read without asking every copy every time. It is a real saving, and it is not free: the write waits for the second-fastest copy, the read waits for the second-fastest answer, and the slow copy sets your tail latency more often than you would like.

Quorum intersection: a write confirmed by two of three replicas and a read of two replicas must overlap on at least one replica

Push the dial the other way and the trade shows. W = 1 and R = 1 is fast and available: a write is done after one copy, a read asks one copy, and stale reads are routine. W = 3 makes every write wait for every copy, so one slow or dead copy blocks writes entirely. There is no setting that is fastest, always available, and always fresh. There are only settings, chosen per kind of data, with the arithmetic written down.

One honest caveat, because the arithmetic has a silent assumption. The overlap proof needs the W confirmations to come from the item's designated replicas, and the R answers to come from that same set. The Dynamo design, where this N, W, R language comes from, deliberately breaks that assumption when replicas are unreachable, and the break has a name: sloppy quorum. A write that cannot reach enough designated replicas is accepted by stand-in machines outside the replica set, and those stand-ins count toward W. Now recompute. The read consults R designated replicas, none of which is the stand-in holding the newest write, and W + R greater than N no longer guarantees any overlap at all. The arithmetic did not fail. Its precondition was quietly swapped.

The stand-in's obligation is called hinted handoff: it keeps the write with a hint naming the replica the write truly belongs to, and delivers it when that replica recovers. If the handoff completes, the system converges and the episode cost only freshness. If the stand-in crashes first, or the hint is lost, the write survives in fewer places than W claimed, possibly nowhere a reader will ever look, and nothing alarms, because every acknowledgment was real. This is why mature quorum systems are explicit about which set they count: Cassandra's quorum levels wait for designated replicas, and a stored hint satisfies only its lowest write level, ANY. When you evaluate a database, the question is not just what W and R are. It is whose names count toward them.

Latency makes the regional version of this concrete. Within one cloud region, a round trip between machines is commonly around 1 millisecond or less to a few milliseconds. Between US regions, expect roughly 60 to 80 milliseconds coast to coast. Between the US and Europe, roughly 70 to 90 milliseconds. Between the US and East or Southeast Asia, roughly 130 to 180 milliseconds. These are planning numbers, not guarantees, and your cloud will differ, but the ratios are stable: coordinating across an ocean costs about a hundred times more waiting than coordinating across a room. A design that needs two cross-ocean round trips before it can answer has a floor near a third of a second before it has done any actual work.

Money follows the same shape. A concrete, documented example: in Amazon DynamoDB, a strongly consistent read of an item up to 4 KB costs twice the read capacity of an eventually consistent read of the same item, 1 read unit against 0.5. The strong read is not a scam or a tax. You are paying for the coordination that makes the freshness promise true. The design question is never strong or weak in the abstract. It is: which reads, in this product, are worth double?

One more cost, easy to miss: availability during trouble. A quorum system with N = 3 and W = 2 can keep writing when one copy is down. It cannot keep writing when two are down, or when a network split leaves a machine unable to reach a majority. At that point a strong system refuses or stalls, which is the correct behavior for its contract, and a very visible behavior for its users. That refusal is the bridge to CAP.

Section 6: CAP, stated honestly

CAP is the most quoted and most misquoted idea in this area, so here it is carefully, in its own terms, and with the usual shortcuts left out.

A network partition is a split where groups of machines cannot talk to each other, even though the machines themselves are still running. CAP says: during a partition, a distributed data system cannot guarantee both of these at once:

Consistency, in the CAP sense: every read receives the most recent write, or an error. This is close to linearizability from Section 3.
Availability, in the CAP sense: every request to a non-failing machine receives a non-error response.

Partition tolerance is not really a third option you can decline. If your system runs on more than one machine, partitions will happen at some point: a cable, a switch, a cloud network incident, a bad deploy. So the practical choice CAP forces is narrower and harder than the slogans suggest: when a partition happens, do you answer with possibly stale data, or do you refuse to answer? A consistency-first system refuses on the side that cannot prove freshness. An availability-first system answers on both sides, accepts that the sides will diverge, and reconciles when the partition heals.

Three corrections to the folklore, because each one changes real designs:

CAP is about the partition window, not normal operation. When the network is healthy, a well built system can be both fresh and available. CAP does not say you must live with stale data forever. It says you cannot have both promises at the exact moment the network splits.
CAP is per operation and per data item, not per company. The same product can refuse writes for a balance during a partition and keep serving a feed with stale data. Drawing one circle around a whole architecture labelled CP or AP is usually a sign the drawing stopped too early.
CAP says nothing about latency when there is no partition. That gap has a companion framing, often written PACELC: if there is a Partition, choose Availability or Consistency; Else, in normal operation, choose Latency or Consistency. PACELC is the more useful daily tool, because most days have no partition, and the latency-versus-freshness trade from Section 5 is the one you actually feel.

During a partition, neither choice is the virtuous one. Refusing protects users from acting on false data, and it turns a network fault into a visible outage. Answering keeps the product up, and it creates conflicting writes that someone, or some code, must merge later. Which harm is worse depends entirely on what the data does. A seat map and a photo feed should not make the same choice, and in Section 8 they will not.

Section 7: The anomalies: what weak guarantees actually look like

An anomaly is a specific forbidden surprise that a weak contract allows. Naming them matters, because in an incident review, stale data is a shrug and a named anomaly is a diagnosis.

Stale read. A read returns an older value after a newer write completed. The profile name story. Harmless for a view count, serious for a permission change: a revoked access token that still reads as valid for a minute is a stale read with a security report attached.

Lost update. Two writers read the same value, each updates it based on what they read, and the second write overwrites the first without ever seeing it. A counter at 10, two workers each add 1 based on the 10 they read, and the final value is 11 instead of 12. The fixes are mechanical once named: atomic increments, compare-and-swap on a version number, or a transaction that retries when the version moved.

Read skew. A reader sees two related items from different points in time, and the pair never existed together. A dashboard reads a total from one snapshot and the line items from a moment later, and the total does not match the items. Nobody wrote bad data. The reader assembled a world that never was.

Write skew. Two writers each check a rule against a snapshot, each write is legal alone, and together they break the rule. The classic shape: at least one doctor must be on call. Two doctors, each seeing the other still on call, each remove themselves. Each check passed. The ward is now uncovered. Write skew is the anomaly that surprises experienced engineers, because no single write was wrong, no data raced on the same row, and a lock on one row would not have saved it. It needs a stronger isolation level, a materialized rule, or a single writer for the rule.

Phantom and ordering surprises. A repeated query gains or loses rows between runs because a concurrent write landed in between, or a feed shows a reply before the post it answers. The second is the causal violation from Section 3 wearing work clothes.

Notice the pattern. Each anomaly is prevented by a specific stronger promise, and each stronger promise has a specific price from Section 5. That is the whole engineering method in this area: name the anomaly your feature cannot tolerate, buy exactly the guarantee that forbids it, and refuse to pay for guarantees whose anomalies nobody would ever notice.

Section 8: Choosing per operation: the decision framework

Work through these questions in order, for the operation in front of you, not for the system as a whole.

What is the worst user-visible outcome of a stale or out-of-order read? If the answer is embarrassment or a refresh fixes it, eventual is fine. If the answer is money moved twice, a seat sold twice, access granted after it was revoked, or a safety rule broken, you need a strong guarantee for that operation.
Does the writer need to see their own write immediately? If yes, buy read your writes at minimum. It is cheap compared to full strong consistency, and it removes the single most reported class of bug in this area: I saved it and it vanished.
Can two writers update the same item at the same time, in ways that must both survive? If yes, plan the merge before you ship: version numbers, atomic operations, or a data type designed to merge. If no merge is acceptable, you need coordination on writes, and you should say so plainly.
How far apart are the copies that must agree? Strong within one region is affordable for many workloads. Strong across continents is a latency budget decision that belongs in the product requirements, not discovered in a load test.
What should happen during a partition? Write the answer down before the partition writes it for you: refuse, serve stale, or serve stale with a visible banner and a queue of writes to reconcile. All three are legitimate. Undecided is not.

A worked table, the kind you can adapt in a design review:

Operation	Sensible default	Why
Password or permission change	Strong, or read your writes plus no stale auth checks	A revoked permission that lingers is a security incident
Account balance and ledger reads	Strong for the decision path	Money decisions cannot be made on a value that may be old
Inventory count shown on a product page	Eventual, with a strong check at checkout	Browsing tolerates staleness; selling does not
Social feed and comments	Causal, plus read your writes for the author	Replies must not float above missing posts
Like and view counts	Eventual	Nobody can tell, or should care, if a count is seconds old
Presence and typing indicators	Eventual, very short lived	The data expires faster than any guarantee matters
Feature flags and configuration	Strong for safety flags, eventual for cosmetic ones	A kill switch that propagates eventually is not a kill switch
Leader election and distributed locks	Strong, quorum based	Two leaders is not a degraded mode; it is corruption in progress

The last column is the deliverable. In a review, the guarantee is a claim. The why is the evidence the claim was thought through.

Section 9: Making eventual consistency safe in the product

Choosing eventual consistency for an operation is not the end of the design. It is the start of a second, product-facing design, because users experience the anomalies, not the model name.

Show the state honestly. If a post is saved on the leader and still propagating, a subtle pending or syncing state is better than pretending the work is instant and global. Users forgive a labelled delay. They do not forgive silent disappearance.

Carry a version, and check it. Every important item should have a version number, a timestamp with care taken about clock skew, or a token that records how far the writer got. Readers and writers can then do compare-and-swap: update this item only if it is still at version 7. If it is at version 8, re-read and decide again. This one habit prevents most lost updates without paying for full strong consistency everywhere.

Give the writer a receipt the reader can wait on. A common pattern behind read your writes: the write returns a position token, sometimes a log sequence number or a version. The writer's next read presents that token, and the replica either has reached that position and answers, or waits briefly, or routes to a copy that has. The guarantee becomes mechanical instead of hopeful.

Design the merge before the conflict. Last writer wins is the default merge in many systems: the write with the later timestamp survives, the other is discarded. It is simple, it converges, and it silently deletes data when clocks are skewed or when both writes mattered. For records a human edited in two places, no algorithm knows the right answer: surface both versions and let a person choose. For the data shapes where merging can be mechanical, there is a designed answer, and it deserves more than the sentence it usually gets.

CRDTs: data types that merge by construction. A CRDT, a conflict-free replicated data type, is a data structure whose merge rule is built into the type, so any two copies that have seen the same updates compute the same value, with no coordinator and no discarded writes. The working set is small. A grow-only counter keeps one count per replica; the total is the sum, and merging takes the larger count for each replica, so concurrent increments are never lost. A PN counter runs two grow-only counters, one for increments and one for decrements, and the value is the difference, which is how you get a counter that can go down as well as up without losing concurrent updates. An observed-remove set tags every addition with a unique marker; a removal deletes only the markers the remover had actually seen, so an item added at the same moment someone elsewhere removed it survives the merge. Shopping carts, playlists, and membership lists are usually this set underneath.

CRDT merge: a grow-only counter takes the larger count per replica, and an observed-remove set keeps concurrently added items

CRDTs come in two delivery styles. State-based CRDTs ship their state, or a summary of it, and merge states directly: simpler to reason about, heavier on the wire. Operation-based CRDTs ship the operations themselves, add this, remove that, and require the transport to deliver each operation reliably, sometimes exactly once and in causal order: lighter messages, more machinery underneath. Same convergence promise, different plumbing bill.

Two honest limits. First, the last-writer-wins register, the CRDT-shaped version of the default above, merges by comparing timestamps, which means it quietly re-imports the entire clock problem from the post on clocks, time, and ordering (#20): when two machines disagree about what time it is, the register picks its winner by consulting a referee that is itself unreliable. It converges, and it can converge on the write that actually happened first. Second, and harder: CRDTs converge, but convergence is not correctness. No CRDT can enforce an invariant that spans the whole value, because enforcing one requires some copy to refuse an update, and refusal requires agreement. Two replicas can each accept the same username, so uniqueness is out. Two replicas can each spend the same last ten dollars, so a balance that must never go negative is out. A seat cannot be sold exactly once. Where every merge outcome is acceptable, CRDTs are the best tool in this post. Where some outcomes are forbidden, you are back to coordination, and to the quorum prices in Section 5.

Keep the window small, and say how small. Eventual with no bound is a shrug. Eventual with a measured bound is a service level: replication lag is under 2 seconds at the 99th percentile, alerted at 10 seconds, and these three operations switch to strong reads if lag crosses a threshold. Measure lag the way #9 taught for any other signal: as a percentile, on a dashboard someone opens, with an alert tied to user harm rather than to a round number.

Repair on the read path: read repair. When a read consults several replicas and they disagree, the coordinator already has everything it needs to heal the data it just touched: return the newest value, and write it back to the stale replicas in passing. Cassandra does this on reads; the pattern is older than any one product. The cost profile is what makes it attractive: repair effort lands exactly on hot data, exactly when someone reads it, at the price of a few extra writes folded into a request that was already happening. The blind spot is just as clear. Data nobody reads is never repaired this way, and a read that consults only one replica (R = 1, from Section 5) has nothing to compare, so it can serve a stale value and fix nothing. Read repair keeps the popular shelf tidy. It does not sweep the warehouse.

Reconcile in the background: anti-entropy. Anti-entropy is the unglamorous name for copies comparing notes outside the request path, so that cold data converges too. The efficient version hashes the data into a Merkle tree: two replicas compare tree roots, walk down only the branches that differ, and exchange just the records under the mismatching leaves, so a comparison costs a few hashes when the copies already agree and targeted transfers when they do not. Its cost profile is the mirror image of read repair: a standing tax of hashing, comparison traffic, and disk reads, paid whether or not anyone ever looks at the data, in exchange for drift anywhere being found and fixed on a schedule. Combined with idempotent writes (#6), the two repairs together are what turn eventually into a promise an operations team can sign: read repair for what users touch, anti-entropy for everything else, and the lag metric from earlier in this section to prove both are keeping up.

Section 10: The principal take: bounded, tested, and per operation

At staff and principal level, the work stops being vocabulary and becomes three disciplines.

First, bound everything you call eventual. The sentence that earns trust in a review is not eventual consistency. It is: eventual, bounded at 2 seconds at the 99th percentile in one region and 5 seconds across regions, measured on the replication lag metric, with these two operations degrading to strong reads when the bound is breached. That sentence can be tested, alerted, and priced. The word eventual alone can be none of those.

The products already sell the bounded version, and their menus teach the traps. Cassandra lets you pick a consistency level per query, and its quorum levels are the arithmetic of Section 5 with regional fine print. In a cluster with three replicas in each of two regions, plain QUORUM means a majority of all six replicas pooled together, four of six: at least one acknowledgment must cross the distance between regions on every operation, and losing either region makes a pooled quorum unreachable. LOCAL_QUORUM means a majority inside the coordinator's own region only, two of the local three, which is what most teams actually meant when they said quorum, and it is the standard production default for a reason. EACH_QUORUM demands a majority in every region before answering, so the slowest or sickest region gates, or outright blocks, writes for everyone; it exists for the rare invariant that must hold globally before anyone is told success, not as a stronger setting to reach for by default. Azure Cosmos DB sells the ladder of Section 3 as five named levels: strong, bounded staleness, session, consistent prefix, and eventual. Its session level, the default, is the session-guarantee bundle from Section 3 sold under its own name, session token and all. And its bounded staleness is this section's sermon with two knobs: reads may lag writes by at most K versions of an item or T seconds, whichever limit is hit first. The floors are documented, and they are instructive: a single-region account can promise as tight as 10 versions or 5 seconds, while a multi-region account cannot promise tighter than 100,000 versions or 300 seconds, because across regions, physics and failure modes will not sign for less. When a vendor sells you a bound with a floor, they are telling you what bounded honestly costs.

Second, test the promise the way an adversary would. Consistency bugs hide in partitions, pauses, and retries, which is why normal testing rarely finds them. The serious practice, popularized by the Jepsen project, is to run a system under deliberate faults: split the network, pause processes, skew clocks, and then check the recorded history of operations against the model the system claimed. If the history cannot be arranged into the promised order, the promise was false, no matter what the documentation said. You do not need that full apparatus for every service. You do need the mindset: a consistency claim that has never survived a partition test is a hope.

Third, expect strong consistency to get cheaper, and buy it deliberately anyway. Modern systems have narrowed the old excuses. Google built Spanner to give externally consistent transactions across continents, paying for it with specialized clock infrastructure and commit waiting, and its team has argued publicly for choosing strong consistency whenever possible because it makes application code simpler and safer. That argument is strong where the budget allows it: every anomaly in Section 7 that a strong model forbids is an anomaly your application code does not have to handle. The principal judgment is knowing where that simplicity is worth the latency, the capacity cost, and the partition behavior, and where a bounded eventual design with a good merge is the honest, cheaper, and perfectly safe answer.

Underneath all three is the line to put on the whiteboard: consistency is a per-operation promise about surprises, with a price in waiting, and a bound you can measure. A design that names the promise, the price, and the bound for each important operation is a design. A design that writes the word consistent once, at the top, is a wish.

Same data, different answers, distilled

The core fact: once data has copies, the copies will sometimes disagree briefly. Consistency is the promise about that window: how fresh, how orderly, and for whom.
Not correctness: a consistent system can store a wrong value, as long as everyone sees it in the promised order. Business correctness is validation and transactions (#15).
The ladder: eventual, then read your writes, monotonic reads, causal, sequential, and linearizable. Each rung forbids more surprises and costs more coordination.
Session guarantees, bought as a bundle: read your writes, monotonic reads, writes follow reads, monotonic writes. Scoped to one client's session and enforced with a token, they remove most "the app feels broken" complaints without global coordination.
Two strong words, not synonyms: linearizable means one item behaving as if there were one copy, in real time. Serializable means multi-item transactions equivalent to some serial order, with no promise about real time. Strict serializability is both, and it is what Spanner sells.
Quorum math: with N copies, a write confirmed by W and a read consulting R overlap when W + R is greater than N. N = 3, W = 2, R = 2 is the common strong-looking setting. The math assumes designated replicas; sloppy quorums and hinted handoff void it by letting stand-ins count toward W.
The price: coordination across a room is about a millisecond; across an ocean it is 130 to 180 milliseconds a round trip. Strong reads can cost double the capacity of eventual reads, as DynamoDB pricing shows for a 4 KB item.
CAP, honestly: during a partition you choose stale answers or refusals. Partitions are not optional. In normal operation the daily trade is latency against consistency, the PACELC framing.
Name the anomaly: stale read, lost update, read skew, write skew. Buy exactly the guarantee that forbids the anomaly your feature cannot tolerate.
Choose per operation: strong for permissions, balances, locks, and kill switches. Eventual, with a bound and a merge plan, for counts, feeds, and presence.
Product menus: Cassandra's LOCAL_QUORUM is the multi-region default teams usually mean; EACH_QUORUM lets the sickest region gate writes everywhere. Cosmos DB's bounded staleness is eventual with a number: at most K versions or T seconds behind.
Make eventual safe: honest pending states, version checks, read-your-writes tokens, merges designed before conflicts, measured lag, and background repair.
Merges have limits: CRDTs (grow-only and PN counters, observed-remove sets) converge with no coordinator, but no CRDT can enforce uniqueness or a never-negative balance; invariants need a copy that can say no. A last-writer-wins register converges by trusting timestamps, which re-imports the clock problem (#20).
Repair in two places: read repair fixes hot data in passing, on the read that noticed it. Anti-entropy, Merkle tree comparisons in the background, covers the cold data read repair never sees. Each pays where the other is blind.
The principal line: bounded, tested, and per operation. Eventual with a number, a metric, and a partition test is engineering. Eventual as a single word is a hope.

Same Data, Different Answers: Consistency, Explained Like You're New

Section 1: The update that went backwards

Section 2: What consistency actually promises

Section 3: The spectrum, in plain language

Section 4: Linearizable is not serializable: the two strong words

Section 5: Why strong costs, and the quorum shortcut

Section 6: CAP, stated honestly

Section 7: The anomalies: what weak guarantees actually look like

Section 8: Choosing per operation: the decision framework

Section 9: Making eventual consistency safe in the product

Section 10: The principal take: bounded, tested, and per operation

Same data, different answers, distilled

Further reading

Comments

Core Concepts

More from this blog

One Leader, Many Witnesses: Consensus and Leader Election, Explained Like You're New

The Log That Remembers: Event Streaming and Pub-Sub, Explained Like You're New

From One Timeline to 500 Million Tweets a Day: Designing Twitter/X in 11 Steps

The Database Menu: SQL vs NoSQL, Explained Like You're New