# Whose Turn Is It: Coordination and Distributed Locks, Explained Like You're New

Two copies of the same scheduler are running, which is exactly what you wanted. One copy was supposed to be insurance for the other. At midnight both copies wake up, both look at the same list of jobs that are due, both correctly decide that the nightly billing job has not run yet, and both run it. Every customer is charged twice. Nothing crashed. No code was wrong in the simple sense: each scheduler did precisely what its instructions said. The missing piece was not logic. The missing piece was agreement about whose turn it was.

That missing piece has a name: coordination. Coordination is how a set of machines that cannot fully trust the network, each other's clocks, or each other's health still manage to act as if they share one brain for a small number of critical decisions: only one of us runs this job, only one of us is the leader, this configuration value is the current one, this worker owns this shard. The distributed lock is the most famous coordination tool, and also the most misused one. Used well, it prevents real damage. Used casually, it creates a confident feeling of safety while a paused, half dead, or partitioned former holder quietly writes over the work of its replacement.

If you are new to this, Sections 1 and 2 need no background beyond knowing that services run on more than one machine. They explain the problem in plain language and then the hard limit underneath it: in an asynchronous network you cannot reliably tell a dead process from a slow one, and every coordination tool is built on top of that uncomfortable fact. Sections 3 through 6 are the working machinery: what a lock actually is, why almost every real lock is a lease with an expiry time, why a lease on its own is not enough and needs a fencing token that the storage checks, and how the coordination services you will actually run, ZooKeeper, etcd, Chubby, and Consul, package these ideas into sessions, ephemeral nodes, and watches. Sections 7 through 9 cover the contested and the broader territory: Redis locks and the Redlock debate, leader election as the most common coordination job in production, and the quieter coordination work of configuration, naming, and membership that causes outages when it goes wrong. Sections 10 and 11 are the failure reality and the judgment calls: how coordination systems themselves fail, how locks get misused in ways that create the outage they were meant to prevent, and a decision framework for the many cases where the correct answer is to design the lock away entirely.

This post builds on four earlier ones and does not repeat them. Consensus (#19) covered how a quorum agrees on a value and elects a leader inside a database or coordination cluster. Clocks (#20) covered why wall clocks cannot be trusted to order events across machines. Failure Detection (#22) covered why deciding that a peer is dead is a probabilistic judgment, never a certainty. Idempotency (#6) covered how to make a repeated operation safe. This post is where those four meet practice: a lock is a failure detector, a clock, and a consensus decision, all pretending to be a simple yes or no, and idempotency is often the safety net that saves you when the pretense breaks.

---

## Section 1: The double run problem, in plain language

Start with one machine. On one machine, taking turns is easy. An operating system gives you mutexes, threads wait in line, and a process that dies takes its locks with it in a way the kernel can clean up deterministically. A cron job on a single server runs once because there is only one cron. A counter in a single process is correct because only one thread increments it at a time under the lock. The whole arrangement works because there is a single, trusted referee with a complete view: the kernel knows which processes are alive, and shared memory does not lie, delay, or partition.

Now run two machines, because one machine might die and you want the service to survive. The referee disappears. There is no shared memory. There is only a network, and the network offers three unpleasant possibilities for every message: it arrives late, it arrives out of order, or it does not arrive at all. Each machine sees only its own local view. Scheduler A cannot look inside Scheduler B. A can only send B a message and interpret silence, and silence is ambiguous in the worst possible way. Silence can mean B is dead. It can also mean B is alive, busy, paused by its runtime, stuck in a long garbage collection, waiting on a slow disk, or separated from A by a broken network path while still happily talking to everyone else, including the database both of them write to.


That ambiguity is the entire problem, and everything else in this post is a response to it. Anytime work must happen exactly once across machines, anytime exactly one writer may touch a resource, anytime a fleet needs exactly one leader, you are asking a set of machines with partial views to agree on a single fact and then to keep honoring that agreement while machines pause, crash, and lose contact with each other. Examples are everywhere once you look. A background job fleet must run each scheduled job once. A payment worker must not settle the same batch twice. A database cluster must promote exactly one replica when the primary dies. A stream processing system must assign each partition to exactly one consumer at a time, or at least know when that assignment has moved. A deployment system must not let two releases roll out over each other. None of these are exotic. They are the daily plumbing of any backend that runs more than one copy of anything.

Notice what the naive fixes do. A boolean flag in the database, set to running before the job starts, looks like a lock and fails like one too: if the holder crashes after setting the flag, the job never runs again until a human clears it, and if two workers read the flag at the same moment before either writes, both proceed. A timestamp that says the job ran at midnight fails when clocks disagree, and clocks always disagree by a little and occasionally by a lot, as the Clocks post covered. Checking and then acting, as two separate steps, contains a race by construction: the world can change between the check and the act, and at scale, with enough machines and enough jobs, eventually it will, usually at the worst moment and in a way no test reproduced. The gap between check and act is where distributed systems bugs live.

![Two schedulers see the same due job and both run it, charging the customer twice, because no atomic claim decided whose turn it was](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794004646/nbhlwej1oxm4m2yq3nxe.png)

The diagram above is the failure in its simplest form. Both schedulers were correct about the facts. The job was due, and it had not run. What they lacked was not information but exclusion: a way for one of them to claim the job atomically, so that the claim itself, not a prior check, decides who acts. That word, atomically, is the hinge. A coordination service earns its keep by offering a small number of operations that are atomic and ordered even though the system behind them is distributed: create this node only if it does not exist, compare this value and swap it only if it still matches, grant this lease to one session at a time. Everything else, locks, elections, and ownership, is built from those few honest primitives.

One more distinction belongs in this first section, because it decides how careful you need to be for the rest of the post. Martin Kleppmann, in his widely read 2016 analysis of distributed locking, drew a line between two kinds of locks. An efficiency lock exists to avoid duplicated work when duplication is wasteful but harmless: if two workers occasionally process the same report, you lose some compute and nothing else. A correctness lock exists because duplication would corrupt something: double charging a customer, overwriting a newer file with an older one, promoting two database primaries that then diverge. The machinery, the risk tolerance, and the honest marketing are completely different for the two kinds. Much of the pain in this field comes from buying an efficiency lock, at efficiency lock prices and with efficiency lock assumptions, and then quietly relying on it for correctness. Keep the distinction in mind through every section that follows. For each tool, ask which kind of lock it can honestly be.

## Section 2: The governing limit: you cannot tell dead from slow

Every field has a result that sets the ceiling for what engineering can promise, and coordination has a particularly unforgiving one. It comes in two related forms.

The first form is the Fischer, Lynch, and Paterson result from 1985, usually called FLP impossibility. Stated carefully, it says that in a fully asynchronous system, where messages can be delayed by any amount and processes can pause for any amount of time, no deterministic algorithm can guarantee consensus if even one process might crash. The honest reading matters. FLP does not say consensus is impossible in practice: the Consensus post covered Raft and Paxos running in production everywhere. FLP says you cannot have all three at once: full asynchrony, determinism, and a guarantee of finishing. Real systems escape by adding an assumption, almost always a timing assumption. They assume messages usually arrive within some bound, they elect leaders with timeouts, and they accept that during periods when the assumption is violated the system may pause rather than decide. Safety is kept. Liveness is rented from the clock.


The second form is the failure detector version of the same wall, covered at length in the Failure Detection post. A perfect failure detector, one that never declares a live process dead and never misses a dead one, cannot be built in an asynchronous network, because the only evidence available is missing messages, and missing messages prove nothing about the sender. Every real failure detector is therefore a bet with tuned odds: wait longer and you will rarely evict a live peer, but failover takes longer; wait less and failover is fast, but you will sometimes declare a slow, healthy peer dead and hand its work to someone else while it is still working. There is no setting where both errors go to zero. There are only settings where you choose which error you can afford.

A lock is a failure detector wearing a different hat. When a lock service grants a lock to worker B because worker A's lease expired, the service is asserting that A is done. It has no proof. It has an absence: A stopped renewing. A may have crashed, which is the case everyone pictures. A may also be paused, partitioned from the lock service but not from the database, or merely delayed behind a network queue. If A later resumes and acts on its belief that it still holds the lock, you have two holders, and the lock has failed at the one job it existed for. This is not a bug in any particular product. It is the shape of the problem. Any lock with an expiry can be violated by a holder that outlives its expiry in its own mind. Any lock without an expiry can be held forever by a corpse. Pick your failure mode, because those are the options.

![The dead or slow ambiguity: the lock service sees only silence from a holder that stopped renewing, cannot tell a crash from a pause or a partition, grants the lock to a second worker either way, and in the pause world the first worker resumes into a two holder state](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794776074/vo9l9l0bjmr9zkincalw.png)

The CAP theorem, stated honestly rather than as a slogan, frames the same choice at the level of the coordination service itself. When the coordination cluster is partitioned, it must choose. A CP style service, which is what ZooKeeper, etcd, and Chubby all are, chooses consistency: the minority side stops granting locks and stops answering authoritatively, so the system as a whole may become unavailable for coordination rather than risk two majorities granting conflicting locks. That is the right choice for coordination data, and it has a sharp operational consequence that surprises teams the first time they meet it: when your coordination service loses quorum, every service that needs a lock, an election, or a config read from it stalls together. Coordination concentrates correctness, and by the same move it concentrates a specific kind of outage. Section 10 returns to that bill when it comes due.

There is a quieter assumption buried under all of this, and naming it now will pay off in Sections 4 and 7: time. Leases expire according to clocks. Failure detectors fire according to clocks. Election timeouts are clocks. The Clocks post showed that machine clocks are only loosely disciplined, that NTP can step a clock backward or forward, and that a process can experience time in jumps because of pauses that the clock does not record as anything unusual. A lease system assumes that the holder, the lock service, and often the storage each measure the passage of time at roughly the same rate and that a process which has been paused for a minute will notice a minute has passed before it acts again. Usually true. Not always true. The famous locking bugs are almost all found in the gap between usually and always.

So the governing picture for the whole post is this. Coordination is possible and useful, and production systems depend on it every second. It is also built on timed bets about liveness that cannot be made certain. The engineering discipline that follows from that picture is not despair. It is defense in depth: use the lock to make conflicts rare, and then make conflicts survivable when the bet loses, with fencing tokens, idempotency, unique constraints, and storage level checks that do not depend on the holder's beliefs. A lock that is your only line of defense for correctness is a lock you are trusting more than the theory allows.

## Section 3: What a distributed lock actually is

Strip the marketing away and a distributed lock is a small record, held in a system that many clients trust, that says who currently owns a named resource. Acquire means creating that record atomically if nobody else holds it. Release means deleting it, but only if you are still the owner recorded. Hold means keeping the record alive for as long as you are doing the work. The record itself is trivial. Everything hard is in those three verbs under failure.

Ownership needs an identity that cannot be confused. Well built clients generate a long random token for each acquisition attempt and store it with the lock record. Release then becomes a compare and delete: delete the record only if the token still matches mine. Without that check there is a classic bug with a body count: worker A's lease expires, worker B acquires the lock, and then A's delayed release request arrives and deletes B's lock, letting worker C in while B is mid work. Three holders, each produced by correct looking code. The token check is a few lines. Omitting it is one of the most common real world locking errors, and it appears in hand rolled Redis locks with depressing regularity, which Section 7 will cover.


The second design decision is what happens when the holder dies. A lock that is only removed by an explicit release is a lock that a single crash can hold forever, so almost no production system works that way. Instead, locks are tied to a session or a lease: a liveness relationship between the client and the coordination service, maintained by heartbeats. While the client keeps the session alive, its locks and registrations live. When the session ends, by explicit close, by the client dying, or by the service deciding the client has been silent too long, everything the session owned is cleaned up automatically. This is the correct default and a genuinely elegant idea. It converts a crash from a permanent blockage into a delay bounded by the session timeout. It also imports the failure detector problem wholesale, because session expiry is precisely the service deciding that silence means death, with all the ambiguity Section 2 described.

The third decision is fairness and queuing. When ten workers want the same lock, who gets it next, and how do the other nine wait? Polling in a loop is the naive answer and it creates a thundering herd: every release wakes every waiter, nine of them lose and go back to polling, and the coordination service spends its capacity processing failed attempts for hot locks. Better designs queue the waiters in order. ZooKeeper's standard lock recipe, for example, has each waiter create a sequential ephemeral node and watch only its immediate predecessor, so a release wakes exactly one waiter. That detail sounds minor until a hot lock with hundreds of waiters melts a coordination cluster that was sized for occasional elections. How waiters wait is a load question, a fairness question, and a starvation question at once: an unfair lock can let a steady stream of newcomers repeatedly beat a waiter that has been queued for minutes, which is a correct lock by the narrow definition and an operational incident by any honest one.

![A fair lock queue in ZooKeeper style: each waiter creates a sequential ephemeral node and watches only its predecessor, so a release wakes exactly one waiter instead of waking the whole herd to poll and retry](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794775885/svgmbukjppo2uvsvpkzr.png)

Reentrancy is the fourth trap, and it is mostly a trap for people porting single machine intuitions. A thread level mutex is often reentrant: the same thread can acquire it twice and must release it twice. A distributed lock held by a process has no reliable notion of the same caller across requests, retries, and service instances. Most distributed lock implementations are not reentrant, and code that acquires in two layers of a call stack can deadlock against itself, waiting for a lock it already holds until the lease expires. The fix is architectural, not configurational: acquire once, at the outermost layer, pass a capability or context down, and structure the work so the critical section is one clear block with a start, an end, and a time budget.

Finally, scope. A lock protects exactly what every participant agrees it protects, and nothing else. If nine services check the lock and a tenth writes to the same resource without checking, the lock is decoration. If the lock name has a typo in one service, or one team uses a per region lock name while another uses a global one, you have two locks that each look correct in isolation. Coarse locks, one for a whole subsystem, are simple and serialize everything behind them. Fine locks, one per customer or per shard, allow real concurrency and multiply the ways naming, cleanup, and contention can go wrong. There is no free choice here, only a visible one: the granularity of the lock is the granularity of the concurrency you are allowing, and it deserves the same review attention as the granularity of a sharding key, because in effect it is one.

---

## Section 4: Leases: the lock that expires, and why expiry is both the fix and the bug

Because holders crash, real distributed locks are almost never open ended. They are leases: ownership granted for a bounded time, renewable while the holder is alive and working, expiring automatically when renewal stops. The word lease is borrowed from the physical world for a reason. You do not own the apartment. You hold it for a term, you renew before the term ends, and if you vanish, the landlord can let it to someone else without breaking down the door. ZooKeeper expresses the same idea as a session: the session lives while heartbeats flow, and its ephemeral nodes, including lock nodes, vanish when it ends. etcd calls it a lease directly, with a time to live in seconds, kept alive by keepalive requests, with keys attached to the lease deleted on expiry. Consul sessions have a TTL with the same shape. Redis expresses it as a key expiry in milliseconds. The vocabulary differs. The bet is identical: silence for longer than the TTL means the holder is gone, and the resource is free.


Choosing the TTL is choosing your position on the failure detector tradeoff from Section 2, with numbers attached. Take a 30 second lease, a common order of magnitude. If the holder crashes one second after acquiring, the resource sits idle for up to 29 seconds before anyone else can take it, plus the time for a waiter to notice and acquire. For a nightly job, that delay is nothing. For a hot lock guarding a checkout path, 29 seconds of forced idleness during an incident is an outage amplifier. Shorten the lease to 5 seconds and failover is snappier, but now a 6 second stall in the holder, a garbage collection pause, a hypervisor hiccup, a noisy neighbor stealing CPU, a brief network queue, creates a second holder while the first still believes it owns the resource. Lengthen it to 5 minutes and false takeover becomes rare, but real takeover becomes so slow that your high availability story is mostly fiction. There is no correct TTL in the abstract. There is only a TTL matched, on purpose, to how long the protected work takes, how bad a double run is, and how long an idle resource can be tolerated. If the work routinely takes longer than the lease, the design is broken before any failure occurs, and renewal is not a patch for that: renewal assumes the holder is healthy enough to renew, which is exactly what a struggling holder may not be.

Renewal itself has failure modes that surprise people. A holder that renews on a background thread can keep a lease alive while the main thread doing the actual work is deadlocked or wedged, so the watchdog confidently reports health for a process that is doing nothing. The opposite split also happens: the work thread is healthy and mid critical section while the renewal thread is starved, the lease expires underneath live work, and the error surfaces later as a mysterious second holder rather than at the moment of the real fault. Mature lock clients therefore tie renewal failure to the work itself: if keepalive fails or the session is lost, the client must be able to cancel or fence the in flight work, not just log a warning and carry on. A lock you cannot act on losing is a lock that tells you the truth too late to use it.

Now the central failure, the one this whole section exists to make vivid. A process can be paused without being killed, and during a pause it experiences nothing. From its point of view, no time passed. Programming runtimes pause for garbage collection, sometimes for seconds under memory pressure. Virtual machines pause for snapshots, live migration, and host contention. Containers are frozen and thawed. A process can be stopped by a debugger, a signal, or a cgroup CPU limit that throttles it to a crawl. The operating system can deschedule it behind a burst of higher priority work. None of these events notify the process that the world moved on. When it resumes, its local variables still say: I hold the lock. Its lease, measured by everyone else's clocks, expired long ago.

![A lease timeline where the holder pauses past its 30 second expiry, a second client acquires and finishes, and the paused holder then wakes and writes anyway](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794004647/cqmft5ospngmqdaear1z.png)

Walk the timeline in the diagram slowly, because it is the single most important picture in this post. At time zero, client A acquires a lease with a 30 second TTL and begins a write that involves a slow downstream call. At 5 seconds A enters a 40 second pause. At 30 seconds the lock service, correctly following its rules, expires the lease. At 31 seconds client B acquires, does its work correctly, and releases. At 45 seconds A wakes up. Nothing in A's world signals a problem. A completes its write at 46 seconds, overwriting or duplicating B's work. Note that every component behaved as designed. The lock service was right to expire. B was right to proceed. A was not malicious or buggy in any locally visible way. The system as a whole still produced two holders, because the lease mechanism has no way to reach into a resumed process and revoke its belief. That missing reach is not an implementation gap that the next version will close. It is the asynchronous pause problem from Section 2, wearing a clock.

Two practical rules fall out of this picture. First, the holder must check its remaining time defensively and stop early: if the protected operation cannot be guaranteed to finish well inside the lease, with margin for pauses and downstream slowness, it should not start, and it should revalidate ownership immediately before the irreversible step, not only at acquisition. That shrinks the window. It does not close it, because a pause can begin in the instant after the final check. Second, and decisively: if a stale holder waking up and writing would corrupt data, the lease cannot be the last line of defense. You need a mechanism that works at the storage, at the moment of the write, regardless of what the writer believes. That mechanism is fencing, and it is the subject of the next section. Teams that skip it have not made their lock simpler. They have decided, often without noticing, that their lock is an efficiency lock, whatever the design document calls it.

Clock behavior adds one more crack. Lease expiry is judged by the lock service's clock, work deadlines by the client's clock, and write ordering sometimes by the storage's clock. NTP discipline keeps these close, usually within milliseconds to tens of milliseconds on a well run fleet, but close is a statistical statement, and clock steps, leap handling, and virtualized clock drift all produce occasional large excursions. The Clocks post covered this in depth. For leases the implication is narrow and sharp: never build a correctness argument that requires two machines to agree on what time it is to within less than your realistic worst clock error plus your realistic worst pause. If your safety margin is 200 milliseconds and your environment can pause a process for 2 seconds, the argument has already failed on paper, before production gets a vote.


## Section 5: Fencing tokens: making a stale holder harmless

Fencing solves the exact failure Section 4 ended with, and its idea is small enough to state in one paragraph. Every time the lock service grants the lock, it also hands out a fencing token: a number that strictly increases with every grant for that resource, 33, then 34, then 35, never repeating and never going backward. The holder must present its token with every write it makes to the protected resource. The resource remembers the highest token it has ever seen and rejects any write carrying a smaller one. Now revisit the pause scenario. A holds token 33 and pauses. B acquires with token 34, writes, and the storage records 34 as the high water mark. A wakes up, still believing, and presents token 33. The storage compares, sees 33 is less than 34, and refuses. A's belief no longer matters. The stale holder has been fenced out, which is where the name comes from: you do not stop the animal from moving, you build a fence it cannot cross.

![A fencing sequence where client B writes with token 34 and the storage then rejects the woken client A because its token 33 is older](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794004649/ibcatuluhuqxz5jrpquc.png)

The elegance is real, and so are the conditions, because fencing is a contract that three parties must all honor. The lock service must generate tokens that are monotonic for the resource, across grants, across leader changes inside the lock service itself, and across restarts. ZooKeeper offers this naturally through its transaction identifiers and the version and sequence properties of znodes, which is one reason its lock recipes can be built safely. etcd offers it through revisions, the global monotonic counter that etcd assigns to every modification, so a lock acquisition can take the revision of its key as a token. Chubby, Google's lock service, built the same idea in from the start under the name sequencers: opaque byte strings a client obtains when it acquires a lock and presents to servers, which check them before serving a request. That Chubby included sequencers in its original 2006 design, rather than bolting them on after an incident, tells you the designers understood the pause problem from the beginning. A lock service that cannot give you a monotonic token can still give you an efficiency lock. It cannot, by itself, give you a correctness lock.

The storage must actually check, on every protected write, and the check must be atomic with the write. A check performed by the client before writing is worthless: it is the check then act race from Section 1 again. The check belongs inside the storage operation: a conditional write, a compare and swap, an update guarded by a condition that the stored token is less than or equal to the presented one, executed as one atomic step by the system that owns the data. In practice this means a version column or token column on the protected row or object, updated in the same transaction as the data. Databases make this natural, which is one more reason Section 11 will argue for pushing exclusion down into the database when you can. Object stores and file systems make it harder, and some offer no conditional operation strong enough, in which case fencing cannot be completed there and the design must change: route writes through a service that can check, or accept that the lock is advisory for that resource.

Two failure modes of fencing itself deserve the same honesty this post applies to locks. The first is partial adoption. Fencing protects the writes that carry and check tokens. One legacy code path, one admin tool, one backfill script that writes to the resource without presenting a token bypasses the fence completely, and it will usually be run at the worst time, during the incident the fence was built for. Fencing is therefore an interface discipline: the storage should reject writes with no token for fenced resources, not merely reject writes with a stale token, so that forgetting is impossible rather than merely unwise. The second failure mode is token scope confusion. A token is meaningful only for the resource and the lock service epoch that issued it. Tokens from two different lock services, or from one service before and after a data loss and rebuild, are not comparable, and comparing them produces confident nonsense. If you ever migrate lock services or restore one from backup, token continuity is a correctness question to answer explicitly, not an operational detail.

It is worth pausing on how often fencing is missing in the wild. Hand rolled locks in application code almost never implement it. Many managed lock libraries implement the lease and the token generation but leave the storage check as an exercise, which in practice means it does not happen. The result is a large population of systems that believe they have correctness locks and actually have efficiency locks with good branding. The audit question is brutally simple, and every team should be able to answer it in one sentence: when a paused former holder wakes up and issues its next write, what, mechanically, stops that write? If the answer describes the holder noticing, the lease being checked by the holder, or a human intervening, you do not have fencing. If the answer names a token or version checked atomically by the storage, you do.


## Section 6: The coordination services you actually run: ZooKeeper, etcd, Chubby, and Consul

Most teams should not build a coordination service, for the same reason most teams should not build a consensus protocol: the Consensus post showed how subtle quorum agreement is, and a coordination service is a consensus system plus a client protocol plus a session model, each with its own edge cases. The practical question is which existing service to adopt and how its specific model shapes your locks. Four names dominate, and comparing them teaches the whole design space.

ZooKeeper, born at Yahoo and later an Apache project, is the elder. Its data model is a small tree of nodes called znodes, kept strongly consistent by its Zab atomic broadcast protocol. Three znode properties do most of the coordination work. Ephemeral znodes disappear when the creating session ends, which gives you presence and lock cleanup. Sequential znodes receive a monotonically increasing suffix from the parent, which gives you fair queuing and a fencing friendly order. Watches let a client register interest in a node and be notified once when it changes, which gives you efficient waiting without polling. The standard recipes built from these primitives, curated for years in Apache Curator, include locks, leader election, barriers, and queues. ZooKeeper's traps are equally characteristic. Sessions expire based on timeouts negotiated within bounds set by the server: the tick time defaults to 2 seconds, and session timeouts are constrained between 2 ticks and 20 ticks by default, so 4 to 40 seconds unless the deployment retunes it, which surprises teams who ask for a 500 millisecond session and silently receive something longer. And disconnection is ambiguous from the client's side: when contact is lost, the session may still be alive on the server and its locks still held, and only session expiry is certain death, which the client cannot tell apart from a passing network failure while it is cut off. Watches are one time triggers, so a client must re-register after every event, and an event can be missed in the gap if the client is not careful, which is why the recipes exist and why using raw watches for locks is discouraged. And ZooKeeper is unapologetically CP: lose quorum and it stops, by design.

etcd, written in Go by the CoreOS team and now a Cloud Native Computing Foundation project, is the coordination store underneath Kubernetes, which makes it by deployment count the most consequential coordination system in the industry: every Kubernetes cluster's control plane state, and with it the scheduling of a large fraction of the world's containers, rests on etcd's consistency. Its model is a flat key value store with a few powerful additions. Every write receives a cluster wide revision number that only increases, which is a ready made fencing token source. Leases attach to keys with a TTL in seconds and are kept alive by a keepalive stream. Watches are streams rather than one time triggers, and a watch can begin from a historical revision, so a client can catch up on what it missed instead of guessing. Its lock and election primitives live in the concurrency package and client libraries, built on the pattern of creating a key under a lease and ordering contenders by revision. etcd's traps mirror its strengths. Because Kubernetes puts everything through it, etcd performance is dominated by write latency and disk fsync behavior, and a slow disk under etcd produces symptoms across the whole cluster that look like unrelated control plane failures. Lease TTLs are in whole seconds and keepalive handling is the client's responsibility. And watch load is real: thousands of clients watching hot keys can turn a small coordination cluster into a notification bottleneck, a problem Kubernetes itself has had to engineer around with caches and proxies at multiple layers.

Chubby, Google's internal lock service, described by Mike Burrows in a 2006 paper that remains the best single read in this field, is the ancestor both of the above were designed in reaction to and in imitation of. Chubby presents as a lock service with a file system like namespace, typically run as five replicas in a cell serving one datacenter area. Google built it for coarse grained coordination: electing a master for a GFS or Bigtable cluster, storing small amounts of authoritative metadata, and service naming. Two lessons from the paper deserve to be quoted in spirit in every design review. First, Google explicitly advised developers to use Chubby for coarse locks held for minutes or hours, and found that developers kept trying to use it for fine grained, high rate locking it was never meant for, which caused load and misuse problems the service then had to defend against. Second, Chubby included sequencers, its fencing tokens, from the start, and also a lock delay mechanism: when a session ends ungracefully, its locks are not released immediately but held for a grace period, so clients and caches downstream are not told a lock is free the instant its holder merely looks dead. Burrows also recorded the cultural lesson that a reliable lock service changes developer behavior: teams began depending on Chubby for problems that should have been solved by making the operation itself safe, and Chubby outages consequently had a blast radius wider than its own data. Section 10 will pick up that thread.

Consul, from HashiCorp, approaches coordination from service discovery first: a catalog of services and health, with a key value store and sessions layered on top. Its sessions have TTLs and, distinctively, a lock delay, defaulting to 15 seconds, during which a lock released by session invalidation cannot be reacquired. That delay is a deliberate safety margin against exactly the paused holder problem: it gives caches and observers time to notice the invalidation before a new holder starts. It is also a trap for the impatient, who experience it as mysterious slowness in failover tests and sometimes disable it, removing a protection they did not know they had. Consul's broader lesson is that coordination, discovery, and health checking are one system in practice, however separate they look in a textbook, and a failure in health information becomes a coordination failure within seconds.

![A quorum coordination service with sessions, ephemeral and sequential nodes, watches, and small strongly consistent keys at its center](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794004649/gh6blcwv3nhn44ufvbuh.png)


Across all four, the shared architecture in the diagram is the point. A small quorum cluster, usually three or five nodes, holds a small amount of precious data, with very strong guarantees and deliberately limited throughput. Clients maintain sessions and watches. Nobody sensible stores bulk data here, serves user reads from here, or takes a lock per user request through here. Coordination data is measured in kilobytes to megabytes, coordination writes run in the hundreds to low thousands per second for a healthy cluster, a figure set by quorum agreement and fsync rather than by the network, while reads scale far higher because they can be served without consensus, and the systems are happiest when elections and lock grants are rare events relative to the work they gate. If your design needs 50,000 lock acquisitions per second, the design is wrong, not the cluster size: you need partitioning of ownership, which Section 11 covers, so that the lock disappears into the assignment of work rather than being fought over per operation.

One more piece of named production evidence belongs here, because it shows the cost of coordination dependence at industry scale. Apache Kafka originally used ZooKeeper for controller election, broker metadata, and topic configuration. As Kafka clusters grew, that dependence became an operational and scaling burden: an extra quorum system to run, metadata bottlenecks, and slow controller failover on large clusters. The Kafka community designed KRaft, its own Raft based metadata mode, to remove the ZooKeeper dependency entirely, a multi year engineering program whose existence is the clearest possible statement that a coordination dependency is a real architectural cost, worth paying for small authoritative state and worth engineering away when the metadata itself becomes the product's core scaling problem. When you adopt a coordination service, you are making the same tradeoff in miniature. Price it honestly.

---

## Section 7: Redis locks and the Redlock debate: the honest version

Sooner or later someone proposes the simplest lock of all: a key in Redis. The recipe is famous. To acquire, a client sets the key to a unique random token with the NX option, meaning only if it does not exist, and a PX expiry in milliseconds, in one atomic command. To release, the client runs a short Lua script that deletes the key only if its value still equals the client's token, which prevents the delete someone else's lock bug from Section 3. In practice most Java teams never hand roll it: they hold the lock through Redisson, the standard Java client, whose watchdog quietly renews the expiry for as long as the holding process stays alive. As far as it goes, this recipe is correct, and for a large class of real problems it is entirely sufficient. If the lock protects against duplicated effort, if occasional double execution is detected and harmless, if the alternative is running an entire ZooKeeper or etcd cluster for one scheduler, a single instance Redis lock is a reasonable engineering choice. The trouble starts at the boundary between that claim and a stronger one.

![The single instance Redis lock recipe: acquire with SET NX PX and a unique random token, then release with a Lua script that deletes the key only if the stored value still equals your token](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794776075/xeegooifdjb1m1re5hug.png)

The stronger claim arrived as Redlock, a distributed algorithm published by Redis creator Salvatore Sanfilippo, usually called antirez. Redlock extends the single key idea across N independent Redis instances, typically five, running on different machines with no replication between them for the lock keys. A client acquires by attempting the same SET NX PX on a majority of the instances, measures how long the attempts took, and considers the lock valid only if it won a majority and the elapsed time leaves enough validity time inside the TTL after accounting for assumed clock drift. Release deletes the key everywhere. The pitch was appealing: a lock with redundancy, built from a beloved tool, without running a consensus system.

![Redlock acquisition across five independent Redis instances: attempt the same SET NX PX on all five, win at least three, then check that elapsed time and assumed clock drift leave enough validity inside the TTL](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794776076/xe95rwfpx868aoysmchj.png)


In 2016 Martin Kleppmann published a careful critique, arguing that Redlock cannot honestly be sold as a correctness lock. His case rested on exactly the foundations this post has built. First, Redlock's safety depends on timing assumptions, bounded clock drift, bounded pause, bounded network delay, that it does not state loudly and that real processes violate, in particular the garbage collection and scheduling pauses from Section 4. A client can win a majority, pause past its TTL, and write anyway, and Redlock gives the storage no fencing token to reject that write with. Second, Redis persistence and restart behavior can resurrect or drop lock state in ways that break mutual exclusion unless configured with care that many deployments do not apply. Third, if what you need is a system whose correctness rests on consensus about lock ownership, the honest tools already exist and are called ZooKeeper and etcd. Antirez replied in detail, disputing parts of the critique, defending Redlock's model as explicitly assuming bounded drift and arguing that the failure scenarios required unlikely coincidences, and the exchange remains one of the most instructive public arguments in distributed systems because both sides agree on the mechanics and disagree on which assumptions deserve to be called reasonable.

The principal reading of that debate, a decade on, is not that one side won. It is that the debate itself revealed the correct frame, which is Section 1's distinction. For efficiency locking, where a rare double run costs wasted work, Redlock and even single Redis locks are defensible, and their operational simplicity is a genuine virtue. For correctness locking, where a double run corrupts money, files, or database primaries, the deciding question is not how many Redis instances voted. It is the fencing question from Section 5: when the stale holder writes, what stops it? Neither Redis design gives the storage a monotonic token as a native, integrated part of the protocol, so the burden shifts entirely to application discipline that the lock itself does nothing to enforce. There is a second, quieter lesson. Any lock whose safety proof contains the phrase assuming clocks do not jump and processes do not pause for long should be read as an efficiency lock wearing formal clothes, because production processes do pause and production clocks do jump, rarely per machine and routinely per fleet. At a thousand machines, a one in a million per hour pause is a monthly event.

The same section of the design space contains database native locks, which are underrated precisely because they are unglamorous. A relational database already runs consensus free, single node atomicity extremely well. A unique constraint is a lock that can never be violated by a pause, because the check happens in the storage at commit time: inserting a row keyed by job identifier and date makes a double run physically impossible to record twice, whatever the scheduler believed. Transaction level and session level advisory locks, such as PostgreSQL's pg_advisory_lock family, give named mutual exclusion inside a system you already operate, with the caveat that session locks are tied to a connection, and a connection pool can return that connection to a different logical user unless the application is careful about which lock flavor it takes. The DynamoDB Lock Client, a library from AWS, builds leases on a DynamoDB table with a lease duration of 20 seconds and a heartbeat period of 5 seconds by default, and its most famous production use is inside the Kinesis Client Library, where workers use DynamoDB leases to divide stream shards among themselves and track checkpoints. That example is worth studying because it shows the pattern at its best: the lease coordinates who processes which shard, and the checkpoint, stored with the lease record, is what makes a takeover safe, not the lease alone. Coordination decides who should act. Durable state decides what has actually been done. Confusing those two jobs is the root error this post keeps returning to.

In practice, the most common scheduled job lock of all uses none of the systems above. It is a row in a lock table in the application's own database, one row per job, holding an owner name and an expiry timestamp. A worker claims the job with a single conditional update that writes its name and a fresh expiry only if the row is free or its recorded expiry has already passed, and the database's atomicity makes that claim a true single winner decision, with no check then act gap. ShedLock, a small JVM library, productizes exactly this pattern for scheduled tasks, and a large share of real teams quietly run it. The attraction is that it needs no new infrastructure, just a table in a database you already operate. The ceiling is that the lock inherits that database's availability and its clock: a database failover stalls every job it gates, and the expiry means whatever the database's time says it means.

![The database lock row pattern: two workers issue the same conditional update against one row in a lock table, the database commits exactly one claim, and the losing worker skips the run](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794776078/o30odtxqytaj5lw5tcft.png)


## Section 8: Leader election: the coordination job that actually runs production

If you survey what coordination services do all day in real fleets, pure application locks are the minority. The majority job is leader election: choosing, for a named role, exactly one live process to be in charge, and rechoosing promptly and safely when it dies. Stateful systems are built this way because many operations are simply easier, or only possible, with a single writer and a single decision maker, and election is how a replicated system manufactures a single writer without fixing its identity in advance. The database primary whose failover the Replication post covered is an elected leader. So is the controller in a Kubernetes control plane, the scheduler in a job system, the consumer group coordinator in a Kafka like design, and the active node in every active passive pair sold as high availability.

The mechanics repeat Section 6's primitives. Candidates contend to create the same ephemeral record, or to win a lease guarded key ordered by revision or sequence. The winner holds the role only while its session or lease lives, renewing continuously. Followers watch the record, and when it disappears they contend again. Kubernetes exposes this pattern directly to operators: components such as kube controller manager and kube scheduler use Lease objects in the coordination API group, through the client go leader election library, whose defaults are a lease duration of 15 seconds, a renew deadline of 10 seconds, and a retry period of 2 seconds. Those three numbers are a complete failure detector policy in miniature. A leader that cannot renew for 10 seconds must step down voluntarily before its 15 second lease can be taken, which is the system asking the old leader to fence itself, because the rest of the system cannot reach into it and do the job. When you tune those numbers on a real cluster you are not adjusting trivia. You are deciding how long the control plane has no active controller after a failure, balanced against how often a transient stall will cause a needless leadership flap, with all the churn and risk a flap carries.

![The leader election cycle: candidates contend for one ephemeral record, the winner renews its lease while followers watch, a disappearance restarts contention, and every election raises the epoch that writes must carry](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794776079/yyr1tk33obtu8lfz4tpc.png)

Election has two failure shapes, and they are not symmetric. The first is no leader: the role is vacant, work in that role stops, and the system degrades in an obvious, monitorable way. The second is two leaders, split brain at the role level: the old leader, partitioned or paused, has not learned it lost, and the new leader, correctly elected by the quorum, is also acting. Two leaders is worse than no leader by an order of magnitude for any role that writes, because no leader is downtime and two leaders is corruption that must later be reconciled, if reconciliation is even possible. This asymmetry should drive every tuning decision. Bias toward vacancy. A system that occasionally pauses for 15 seconds during a partition is a system with an availability story to improve. A system that occasionally runs two primaries is a system with a data story it may never fully repair. It is the CAP choice from Section 2, felt at the level of a single role: the quorum side continues, the minority side must stop acting, and a leader that cannot confirm it still speaks for the quorum must treat itself as deposed, not as merely uninformed.

![Two leaders after a partition: the majority side elects a new leader at epoch 8 while the deposed leader still acts at epoch 7 on the minority side, and storage accepts the newer epoch and rejects the older one](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794776080/agwdyrfpzfdzciauwog0.png)

Notice, too, what election does not solve. Electing a leader decides who may act. It does not make the leader's actions safe if the leader is stale in other ways: reading old state, replaying old work, or writing over a successor's output during the overlap window before it learns it was deposed. Serious systems therefore pair election with the same defense as locks. The new leader starts from durable state, not from the old leader's memory. Writes carry a term, epoch, or fencing token that increases at each election, the pattern Raft builds in with its terms and that the Consensus post covered, so storage and followers can reject the old term's commands. Observers validate the leader by its epoch, not by its confidence. A deposed leader that still feels like the leader is a normal event in distributed systems, not an anomaly, and the architecture has to treat its output as untrusted until its epoch checks out. Once you see election and fencing as one mechanism viewed from two sides, a lot of seemingly separate machinery in Raft, in Kubernetes, and in database failover collapses into a single idea: authority is a number, the number only moves forward, and everyone checks the number.


## Section 9: The quieter coordination: configuration, naming, and membership

Locks get the papers and the arguments, but a coordination service in a real fleet spends much of its life on three quieter jobs, and outages in each are instructive because they show coordination failing without any lock being involved.

The first is configuration: the small set of values everyone must agree on, feature flags with strong semantics, the current schema version, the list of shards and their owners, rate limit parameters that must move together. Storing such values in a coordination service buys atomic update and watch based notification, so a change lands everywhere as one event rather than as a rolling inconsistency. The failure mode is the mirror image of the benefit. A bad value, pushed atomically and watched eagerly by ten thousand clients, becomes a fleet wide outage at the speed of the watch mechanism, faster and more uniformly than any deployment pipeline could manage. Teams learn, usually once, to treat coordinated configuration with deployment grade discipline: validation before write, staged rollout where the value allows it, a way to pin and roll back, and an audit log of who changed what. The system that makes config consistent also makes config mistakes consistent. The second failure mode is subtler: clients that treat a watch as guaranteed instant delivery and build correctness on instant propagation. Watches are fast and they are not synchronous. There is always a window where some clients have the new value and others have the old, and any design that cannot tolerate that window has mistaken a notification for a transaction.

![Coordinated configuration propagation: one written value fires watch notifications to ten thousand clients at once, which is the benefit when the value is good and a fleet wide outage at watch speed when the value is bad](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794776081/sf3ooy9oi9tccprchwkx.png)

The second job is naming and discovery: which processes provide a service, at which addresses, in which state. Coordination backed registries, ZooKeeper in early Hadoop era systems, etcd behind Kubernetes Services and Endpoints, Consul as a dedicated product, give discovery a consistent source with health attached, rather than a scattering of stale DNS answers. The corresponding failure is health information lying in either direction. Mark a live node dead and traffic drains from capacity you needed. Mark a dead node live, because its session has not yet expired, and clients route work into a hole for the length of the timeout. Discovery is failure detection again, with user visible consequences, and its timeouts deserve the same deliberate tuning as lock TTLs rather than the defaults that shipped in a tutorial.

The third job is membership and group coordination proper: maintaining the agreed list of who is in a group, running barriers where a set of workers must all arrive before any proceeds, and building distributed queues. The Failure Detection post covered gossip based membership, as used by systems that favor availability and eventual agreement about the roster. Quorum coordination services offer the strong version of the same list, at quorum prices. The design question is which version each decision needs. Routing hints can be eventually consistent and gossip is ideal. The list of replicas that hold acknowledged data cannot be a rumor. Mixing the two, using a gossip supplied roster to make a durability decision, or paying quorum costs to distribute a routing hint, are both common and both expensive in opposite currencies.

Underneath all three jobs sits the watch mechanism, and one operational pattern deserves a name because it has caused real load incidents: the herd on a hot key. If many thousands of clients watch the same node and that node changes, the service must deliver many thousands of notifications, and the clients then typically re-read and re-register, arriving as a synchronized burst. Chubby's designers saw this early and answered with lock delay and with guidance to structure data so that hot shared nodes are rare. Kubernetes hit the same shape at larger scale and answered with layered caching, so most watchers are served by an intermediate cache rather than by etcd directly. The lesson generalizes beyond any product: coordination systems are built for small data and rare changes. If your change rate or your watcher count is the interesting number in the design, you have placed a broadcast problem on a consensus system, and it will charge you consensus prices for broadcast work.

![The herd on a hot key: one change delivers thousands of watch notifications, thousands of clients re-read and re-register in a synchronized burst, and production systems answer with layered caching in front of the cluster](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794776083/di7gneodc0rgfmwjxtoh.png)


## Section 10: When coordination itself fails: the failure modes of the solution

A standing rule of this series is that every solution gets its failure modes examined as carefully as the problem it solves. Coordination systems fail in characteristic ways, and because so much depends on them, their failures are rarely local.

Quorum loss is the defining one. A three node coordination cluster tolerates one dead node. Lose a second, to failure, maintenance overlap, or a network partition that isolates each node, and the cluster cannot safely answer. Every dependent service then discovers, simultaneously, how many of its code paths assumed a lock, an election, or a config read would always succeed. Well built clients surface this as explicit unavailability of the coordinated function. Poorly built clients surface it as threads piling up on blocked lock calls, retry storms against a cluster that cannot answer, and cascading exhaustion of connection pools and request queues, the exact overload dynamics the Backpressure post covered, triggered from the control plane outward. At Google, the Chubby paper's observation that outages of the lock service affected many dependents led to a deliberate cultural and technical effort to reduce gratuitous dependence: use Chubby where its guarantees are needed, and design systems to degrade rather than halt where they are not. That is still the correct posture. For every dependency on coordination, write down what happens when coordination is down for ten minutes. If the honest answer is everything stops, that may be acceptable for a small core. It is a finding, not a detail, when the list turns out to include systems nobody thought of as coordinated at all.

![Quorum loss in a three node coordination cluster: once a second node is lost the cluster cannot grant locks or elect leaders, dependent services block and retry together, and the outage cascades from the control plane into the data plane](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794775886/zxiwhwrufek5niw47lw4.png)

Contention collapse is the second failure. A coordination cluster is a small quorum system with fsync bound write latency. Its throughput is measured in thousands of operations per second on a good day, not millions. A design that takes a global lock per request, or that lets hundreds of waiters poll aggressively, or that stores fast changing data in watched keys, can saturate it with traffic that is individually legitimate. Worse, saturation degrades the cluster's own heartbeats and elections, so overload in the coordination plane can flip leaders and expire sessions, which generates more lock churn, which deepens the overload. It is a feedback loop with the same shape as a retry storm, and the defenses are the same in kind: bound the clients, back off with jitter, cache reads with an explicit staleness budget, and above all reduce the coordination rate by redesigning ownership so the common case needs no coordination at all.

![Contention collapse as a feedback loop: client saturation raises write latency, degraded heartbeats and elections expire sessions, expired sessions create more lock churn, and the churn deepens the saturation that caused it](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794776084/biwfauv6mtams52ytoly.png)

Misuse failures form the third group, and they are the most common of all because they need no component to break. Holding a lock across a slow external call, a user think time gap, or an unbounded retry loop converts a short critical section into an open ended one and guarantees eventual lease expiry mid work. Locking at the wrong granularity serializes a system that was built for parallelism, and the symptom, a mysterious throughput ceiling far below hardware limits, can survive for months because every individual lock acquisition looks fast. Deadlock, supposedly a single machine problem, returns in distributed form when two services acquire two locks in opposite orders through different code paths: there is no global deadlock detector to save you, only lease expiry as a crude and slow referee, and the correct fix is the old one, a global acquisition order, or better, never holding two locks at once. Lock leaks through client bugs, where a release path is skipped on an error branch, are bounded by leases but still produce periodic stalls that correlate with error rates in a way that baffles the first investigation. And the subtlest misuse of all is the lock as a substitute for a data design: teams lock around a read modify write on a shared counter, a shared file, or a shared balance for years, paying coordination costs and accepting coordination risks, when a schema change, an append only log of events, a unique constraint, or a sharded ownership model would have removed the shared mutable state that made the lock necessary. The lock worked, which is exactly why the underlying problem never got fixed.

Testing deserves its own paragraph in this section, because coordination bugs are disproportionately bugs that staging cannot see. A lock implementation can pass every functional test and still be unsafe, because its safety depends on pause, partition, and clock behavior that a healthy test environment does not produce. The serious teams in this space test with fault injection as a first class practice: pausing a holder past its lease with a stop signal, partitioning the holder from the lock service while leaving its path to storage open, stepping clocks, and killing coordination nodes during elections. The Jepsen project, which subjects distributed systems to exactly this class of adversarial testing and publishes the results, has found correctness violations in system after system that believed its coordination was sound, and its reports are required reading for anyone choosing a coordination primitive for correctness work. If your lock has never been tested against a paused holder, you do not know whether you have a correctness lock. You have a hope with an API.


## Section 11: The principal take: earn the lock, or design it away

Here is the judgment this post has been building toward. A distributed lock is not a building block to reach for early. It is a cost center you adopt when you have proven that the cheaper, stronger alternatives do not fit, and even then it is only one layer of the defense. The decision order below is the one to apply in design review, in sequence, stopping at the first step that honestly works.

![A decision flow that starts by assigning a single owner, then separates efficiency locks from correctness locks, and requires fencing or storage level protection for correctness](https://res.cloudinary.com/i5oc7fbp/image/upload/v1794004650/zg218qrz7argujqxjsuy.png)

First, try to make the question disappear by assigning ownership. If work can be partitioned so that each key, shard, customer, or partition has exactly one owner at a time, and ownership changes are rare and deliberate, then there is no contention to lock against in the common case. This is why stream systems assign partitions to consumers, why sharded databases route a key to its shard, and why the Sharding post's ownership map is quietly a coordination decision. Consistent hashing, shard maps stored in a coordination service, and queue per aggregate patterns all buy the same thing: mutual exclusion by construction, paid for once in routing rather than per operation in locking. Most systems that think they need a lock per request actually need an owner per key.

Second, try to make a double run harmless. Idempotency keys, deduplication tables, unique constraints, and compare and swap updates all convert a correctness panic into a manageable retry, as the Idempotency and Distributed Transactions posts covered in depth. If processing the same job twice is safe because the second run is detected and absorbed at the storage, then your lock is demoted to an efficiency lock, and a simple one will do, because its rare failure costs a little duplicated effort that the storage then discards. This is the best possible outcome for a lock: present, useful, and not load bearing for correctness. Design toward it deliberately. Ask of every locked operation not how do we guarantee it runs once, but how do we guarantee its effect happens once even if it runs twice. The second question has answers that survive pauses and partitions. The first, strictly, does not.

Third, if exclusion must be enforced outside the storage, enforce it in the storage anyway on the way through. That sentence is the fencing section compressed. Use the coordination service lease for what it is good at, keeping double runs rare and failover automatic, and require the monotonic token check at the point of write for what the lease cannot do, stopping the stale holder. If the storage cannot check a token or a version, change where the write goes, or accept in writing, in the design document, that the lock is advisory for that resource. Unwritten acceptance is how efficiency locks get promoted to correctness locks by accident.

Fourth, when you do run a coordination service, run it like the critical, small, boring infrastructure it is. Three or five nodes, dedicated or carefully isolated, on fast disks, with its own monitoring for leader changes, session expirations, proposal latency, and quorum health, and with client libraries, Curator for ZooKeeper, the official etcd clients, the Kubernetes leader election library, rather than hand rolled recipes. Pin session and lease timeouts to a written rationale per use case, not a global default nobody chose. Load test the lock paths you depend on and fault test the paused holder case before production tests it for you. Keep bulk data, hot keys, and per request locking out of it. And write the degradation contract: which product features pause when coordination is unavailable, and which must never have depended on it in the first place.

When not to use a distributed lock at all, stated plainly. Do not use one to protect a resource that some writers can reach without it. Do not use one across a human timescale interaction. Do not use one as a rate limiter, a queue, or a workflow engine: each of those has a proper tool, covered in the Rate Limiting and Async Processing posts, and a lock pressed into their service fails in ways none of them would. Do not use a Redis lock, single or Redlock, as the sole guard on money movement, file correctness, or primary election in a system whose data you cannot reconcile afterward. Do not hold one while calling a service you do not control. And do not add one to a design to avoid a schema change you know is right: the lock will still be there, expiring and flapping and demanding care, long after the schema change would have been forgotten.

The closing thought is about what coordination teaches when you study it honestly. Single machine programming trains an intuition that exclusion is a fact: hold the mutex and the world waits. Distributed coordination replaces that fact with a lease, a bet, and a fence: authority that expires, granted on timed evidence, and rechecked at the point where damage would occur. Systems built with that humility, ownership first, idempotent effects second, fencing third, and locks as the rare exception rather than the default glue, are the ones that survive their own success. The question in the title, whose turn is it, turns out to have a principal level answer. Design so that the turn belongs to someone by construction, make acting twice survivable when construction fails, and reserve the lock for the small remainder, with a fence around it. That is the whole discipline, and it is enough.


## Whose turn is it, distilled

* Coordination is agreement among machines that cannot fully observe each other. Silence is ambiguous: a silent peer may be dead, paused, or partitioned, and no mechanism can remove that ambiguity.
* FLP impossibility and the impossibility of a perfect failure detector set the ceiling. Real systems keep safety and rent liveness from timing assumptions: timeouts, leases, and election deadlines.
* A lock is a failure detector in disguise. A lock with no expiry can be held forever by a dead holder. A lock with an expiry can be violated by a holder that pauses past expiry and wakes up still believing. Choose knowingly.
* Separate efficiency locks from correctness locks. A double run that wastes work needs the first. A double run that corrupts data needs the second, and the second requires more than a lock.
* A lease TTL is a policy decision with numbers: short leases fail over fast and invite false takeover, long leases resist false takeover and make real failover slow. Match the TTL to the work, with margin.
* Renewal must be able to stop the work, not just report a problem. A holder that learns it lost the lock after it writes has learned too late.
* Fencing tokens close the pause hole. Every grant carries a strictly increasing token, the holder presents it on every write, and the storage atomically rejects writes with a token below the highest it has seen.
* Fencing fails if any write path skips the check. The storage should reject tokenless writes to fenced resources, and tokens are only comparable within one lock service and one epoch of that service.
* ZooKeeper gives sessions, ephemeral and sequential znodes, and one time watches, with session timeouts bounded by tick based limits. etcd gives revisions as ready made fencing tokens, leases, and streaming watches, and carries Kubernetes on its back. Chubby taught the field coarse grained locking, sequencers, and lock delay. Consul adds a 15 second default lock delay that trades failover speed for safety.
* Redis locks are fine efficiency locks. Redlock's majority across independent instances does not create a fencing token, and its safety rests on clock and pause assumptions production fleets violate. Never let a Redis lock be the only guard on an unreconcilable write.
* Database mechanisms are locks that cannot be fooled by a pause: unique constraints, conditional updates, and advisory locks checked inside the storage. The DynamoDB Lock Client shows the mature pattern: leases assign the work, durable checkpoints record what is done.
* Leader election manufactures a single writer. Two leaders are far worse than no leader. Bias tuning toward vacancy, pair every election with an epoch or term, and make a deposed leader's writes rejectable.
* Coordination also carries configuration, discovery, and membership. A bad config value propagates at watch speed, discovery is failure detection with user visible timeouts, and a hot watched key can herd thousands of clients into a burst.
* Coordination systems fail by quorum loss, by contention collapse, and by misuse: locks held across slow calls, wrong granularity, distributed deadlock, and locks used as a substitute for a data design. Test with paused and partitioned holders, or your safety claim is untested.
* The principal order of operations: assign an owner, make double execution harmless, fence what remains, and only then operate a lock, with a written plan for the day the coordination service itself is down.

*Blueprints of Scale: Core Concepts #25. If this helped, the best thanks is a share with someone who is learning.*

## Further reading

* Mike Burrows, "The Chubby Lock Service for Loosely-Coupled Distributed Systems", the original Chubby paper, on coarse grained locking, sequencers, and lock delay: https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems/
* Martin Kleppmann, "How to do distributed locking", the efficiency versus correctness distinction and the Redlock critique, with the fencing token explanation: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
* Salvatore Sanfilippo, "Redlock: Distributed locks with Redis", the Redlock algorithm in its author's words, best read alongside the critique: https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/
* Apache ZooKeeper documentation and recipes, for sessions, ephemeral and sequential nodes, watches, and the standard lock recipe: https://zookeeper.apache.org/doc/current/recipes.html
* etcd documentation, "Leases" and the concurrency API, for lease TTLs, keepalives, revisions, and lock and election primitives: https://etcd.io/docs/v3.5/op-guide/
* Kubernetes documentation, "Leases" and the coordinated leader election concepts, for Lease objects and how control plane components elect a leader: https://kubernetes.io/docs/concepts/architecture/leases/
* AWS Builders' Library and the Amazon DynamoDB Lock Client documentation, for lease based shard coordination as used with Kinesis: https://github.com/awslabs/amazon-dynamodb-lock-client
* Apache Kafka documentation on KRaft, for why a major system engineered its way off an external ZooKeeper dependency: https://kafka.apache.org/documentation/#kraft
* Jepsen analyses index, for adversarial testing results on coordination and storage systems under pauses and partitions: https://jepsen.io/analyses
* Google SRE Book, "Handling Overload" and the chapters on distributed consensus, for how these mechanisms behave under production load: https://sre.google/books/

