Here's what's covered: the 3 AM story, why green dashboards lie, and the difference between watching from inside and watching from the customer's seat; the three pillars (metrics, logs, traces), what they leave out, and the one thing that actually matters, which is that they can be joined; why averages lie and what percentiles promise; how to compute a percentile across five hundred servers without lying (histograms, and what's replaced fixed buckets); distributed tracing: spans, context propagation across services and queues, and the sampling decision; structured logging, cardinality, and what never goes in a log; RED, USE, the four golden signals, and dashboards people actually open; alerting: symptoms not causes, SLOs and error budgets, multi-window burn rates, runbooks, on-call, and postmortems; and the principal-level toolkit: what tails do in deep call graphs, the telemetry bill and its levers, cardinality as a budget, and game days.

If you're new to this, Sections 1 and 2 need nothing but the memory of one bad night. Sections 3 through 8 are the machinery every backend engineer meets in production. Section 9 is the judgment. The cheat sheet is at the end under What Broke at 3 AM, distilled, and every diagram is also described in the text around it.

Section 1 — The 3 AM page

Friday night, peak dinner rush. The on-call phone buzzes: "checkout is slow." Not down, just slow.

The on-call opens the dashboard wall. Request rate: normal. Error rate: 0.1%, normal. CPU across the fleet: 40%, normal. The database: "healthy." Everything the team chose to watch is green, and customers are abandoning carts. Forty minutes in, someone thinks to check the payment provider's status page: degraded, elevated latency on their API. The slowness was never inside the company's systems at all. It was a dependency, and no dashboard watched it, because nobody had asked the question "what does slow look like from the customer's seat?"

There are two lessons here, and the obvious one is fine as far as it goes: every external dependency you rely on gets its own latency and error graph, because "the payment provider is slow" is a question you'll ask again. Google's SRE book distinguishes white-box monitoring (signals from inside the system: CPU, queue depth, your own request timings) from black-box monitoring (probing the system from outside, the way a user would), and this team had only the first kind. Two cheap tools give you the second. Synthetic monitoring runs a scripted checkout from a few regions every minute and alerts when it slows, which would have caught this in the first minute. Real user monitoring (RUM) puts a small script in the page to report what actual customers experienced, including the network and the browser, which server-side timings never see.

The bigger lesson is the one this post is about. Monitoring answers the questions you thought of in advance, and incidents ask questions you didn't. The team had monitoring: dozens of graphs, all green. What they lacked was the ability to ask a new question at 3 AM ("show me checkout latency broken down by downstream dependency, for the last 20 minutes") and get an answer in seconds. Charity Majors, who did more than anyone to make this distinction popular, frames it as known unknowns versus unknown unknowns: monitoring tells you when something you predicted is wrong; observability lets you figure out something you never predicted. (The word itself is older; it comes from control theory, where it means being able to infer a system's internal state from its outputs.) You need both. But teams chronically over-invest in the first (more dashboards, more graphs) and under-invest in the second: the high-cardinality data, the request-scoped context, the tooling that lets a tired human slice production a new way.

The 3 AM test is this post in one sentence: at 3 AM, with a symptom nobody predicted, can you find the cause in under ten minutes? If not, what you have is not an observability gap but an observability absence.

Section 2 — The three pillars (and what you actually need)

Every observability vendor will teach you the three pillars. The framing comes from a 2017 essay by Peter Bourgon (who drew it as a Venn diagram organized by what each signal is good for, not as a checklist), and Cindy Sridharan's 2018 report made it standard:

• Metrics: numbers over time. Request rate, error rate, CPU, queue depth. Cheap to store, cheap to graph, the backbone of dashboards and alerts. A metric answers "how much, how fast, how many." (Two shapes: a counter only goes up, like requests served; a gauge goes up and down, like queue depth.)
• Logs: discrete event records. "At 03:07:12, request abc-123 failed with timeout after 30s." The narrative record: what happened, in words.
• Traces: the journey of one request across services, as a tree of timed steps called spans. Request abc-123 spent 12 ms in the gateway, 400 ms in the checkout service, 2.1 s waiting on the payment provider. Traces answer "where did the time go?"

The pillars are a useful vocabulary and a poor design principle. Nobody at 3 AM thinks "I need a pillar." They think "why is this customer's checkout slow and nobody else's?", and answering that needs all three at once: the metric that shows the spike, the trace that shows where the time went, the log that shows what the payment provider actually said. What you actually need is not three data types. It's the ability to ask a new question about production and get an answer fast. Metrics, logs, and traces are the three shapes the answers come in.

Here's the failure that proves it. A team buys the full platform, metrics and logging and tracing, and at the next incident still takes an hour. The postmortem finds three disconnections: their traces sample 1 in 1,000 requests (so the slow ones are never captured), their logs have no request ID (so logs can't be joined to traces), and their metrics are rolled up to one-minute averages (so a 20-second spike is diluted into a bump nobody notices). They own all three pillars and can't answer one question, because the pillars aren't connected. The fix is three decisions, not more tooling: sample traces intelligently (Section 5), put the request ID in every log line (Section 6), and keep metrics at fine granularity (Section 4).

Learn this phrase, because it recurs all through the post: the trace ID is the thread that stitches the three pillars together. One ID, generated at the edge, propagated through every service, attached to every log line and every span. Note what's not in that list: metric labels. A metric label with a distinct value per request creates one time series per request, which is the cardinality explosion Section 6 is about. The mechanism that links metrics to traces is the exemplar: a sampled trace ID attached to a histogram bucket, so that when you click the p99 spike on a graph, the dashboard can hand you an actual slow request from that bucket. With the ID everywhere it belongs, the 3 AM question becomes mechanical: find the spike in the metric, follow its exemplar to a trace, read that trace's logs. Without it, you have three separate haystacks.

Two signals the three-pillar picture leaves out, both worth knowing by name. Continuous profiling samples where the CPU and memory are actually going inside a process, all the time, at low overhead; it answers "why is this service hot" at a level of detail a trace can't, and OpenTelemetry now treats it as a fourth signal. And eBPF-based instrumentation (Beyla, Pixie, and similar) captures request timings from the kernel without touching application code, which is the answer to "the three internal HTTP clients nobody instrumented" in Section 5.

Observability is not a purchase but the discipline of making your telemetry joinable.

Section 3 — Why averages lie

A thousand requests: 990 at 50 ms, 10 at 7 seconds. The average is 119.5 milliseconds, a lovely number describing an experience nobody had. Nobody experienced 120 ms. Nine hundred and ninety people experienced 50 ms and ten people experienced 7 seconds of staring at a spinner. The average erased the ten who suffered.

Every senior engineer has lived the dashboard version: average checkout latency reads 120 ms, the manager is happy, and support tickets pile up saying "checkout hangs." Both are true. The arithmetic is correct and the number is a lie.

Percentiles are the fix, and the definition is plain: line up all your requests from fastest to slowest. The p50 (the median) is the middle one; half your users were faster, half slower. The p99 is the one 99% of the way down the line: 99% of your users were faster than this, 1% slower. The p99.9 is 99.9% of the way down. (You'll see "p999" in some tools; it's the same thing.) In the example, the p99 is still 50 ms by the usual nearest-rank definition, because the ten slow requests are the last 1% exactly (a tool that interpolates between neighbors would report something higher, since the tail starts right at the boundary; the point survives either way), and the p99.9 is 7 seconds. A percentile is a promise with a named exception rate: "p99 latency of 300 ms" means "all but 1 in 100 requests finished within 300 ms," and it names exactly how many users you're allowed to disappoint. One footnote for when you compute these yourself: with few samples, different tools interpolate differently, and "the p99 of 100 requests" can come out as the 99th value or as something between the 99th and 100th. With thousands of samples it stops mattering.

The intuition that makes it stick: at scale, the tail is not rare. One percent sounds tiny until you multiply. A service handling 1,000 requests a second at p99 = 300 ms has ten users every second waiting longer than 300 ms. That's 864,000 slow experiences a day, each one a potential support ticket, an abandoned cart, a retry. Averages hide the tail; percentiles price it. This is why every latency promise that matters is written in percentiles: "p99 under 500 ms," never "average under 500 ms." (Contractual SLAs are usually written in availability, like 99.9% uptime; when they mention latency, it's a percentile.)

The practical rule for which ones to watch: p50 tells you about the typical experience, p99 tells you about the broken experience, p99.9 tells you about the pathological one. Watch p50 and p99 on every latency dashboard, always, as a pair. If p50 is fine and p99 is burning, you don't have a capacity problem; you have a some-requests-are-special problem: a slow dependency, a garbage-collection pause, one bad shard, a cold cache. Add p99.9 when you're big enough that 1-in-1,000 events happen constantly; at 10,000 requests a second, the p99.9 fires ten times a second.

And measure from the right place. Server-side latency starts when the request reaches your code and ends when the response leaves it. The customer's latency includes DNS, the TLS handshake, the network in both directions, and the browser rendering the result, and the two can differ by hundreds of milliseconds on a mobile connection. Section 1's synthetic and real-user monitoring are how you see the second number.

The quarter teams lose chasing averages goes like this: tuning, caching, indexing, and the average barely moves, because it's dominated by a long tail of requests hitting one overloaded database replica. The day the dashboard switches to p50 and p99, the problem announces itself: p50 flat and beautiful, p99 spiking every time traffic shifts to the bad replica. Three months spent optimizing a number that couldn't see the problem. The percentile didn't just measure better. It diagnosed.

Averages are fine for things that don't have tails, like a single machine's CPU utilization or a queue's depth, and even there be careful: a fleet-average CPU of 40% hides the one instance at 100%. For anything with a distribution, if someone shows you an average latency graph in a review, ask for the p99. Every time.

Section 4 — Counting the tail

Percentiles sound simple until you have 500 servers. Each one sees only its own requests. To get a global p99, you'd need every latency sample from every server in one place, and while that's storable in principle (10,000 samples a second is around 7 to 14 gigabytes a day, depending on how you store them), the central sort is slow, and it gets slower every time you want the p99 broken down by endpoint and region. So the industry uses approximations, and the difference between them is a real design decision:

• Summaries: each server computes its own percentiles (its local p99) and reports those up. Cheap. But percentiles of percentiles are wrong. The p99 of five servers' p99s is not the fleet's p99, and it's not even close when traffic is uneven, because the quiet server's p99 counts as much as the busy one's. The Prometheus documentation puts it flatly: averaging quantiles "yields statistically nonsensical values." Summaries are fast, cheap, and mathematically dishonest.
• Histograms: each server sorts its latencies into buckets ("how many requests took 0–10 ms, 10–25 ms, 25–50 ms…") and reports the bucket counts. The central system adds the buckets across servers (counts add perfectly), then reads percentiles off the merged histogram. Histograms aggregate correctly, at the cost of precision limited to the bucket width.

Here's what it looks like in the most common tool. A Prometheus histogram named http_request_duration_seconds exposes a series per bucket with a label le ("less than or equal"), cumulative, so le="0.1" counts every request under 100 ms and le="0.5" counts every request under 500 ms including those. The query histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) adds the buckets across every server for the last five minutes and reads off the p99. That sum by (le) is the whole point: it's the merge that summaries can't do.

Use histograms for latency. Two developments have removed the old objection that you had to pick bucket boundaries in advance. Prometheus's native histograms and OpenTelemetry's exponential histograms choose exponentially spaced buckets automatically at a configured resolution, so they cover microseconds to minutes without configuration. And a family of sketches (HDR Histogram, t-digest, Datadog's DDSketch) stores a compact approximation of the whole distribution that merges correctly; DDSketch is the one with a formal relative-error guarantee, HDR Histogram guarantees precision within a configured range, and t-digest is accurate in practice without a formal bound. If you're on classic fixed-bucket histograms, pick boundaries that match the latencies you care about: fine buckets where your target lives (around 100 to 500 ms if that's your promise), coarser buckets out in the tail. And keep the raw buckets, not just the computed percentiles. Buckets let you recompute any percentile later; precomputed percentiles lock you into the questions you asked at collection time. The 3 AM principle again.

Time is another axis you can lie along. A dashboard that stores a p99 per minute and then shows "the p99 for the hour" by averaging sixty values has made the same mistake sideways: the one bad minute is diluted by fifty-nine good ones. Histograms fix this too, because bucket counts add across time as well as across servers.

What a summary-based fleet p99 hides is usually a region. It sits at 200 ms for months while customers complain, and the day histograms replace summaries, the truth appears: one region's p99 is 2 seconds, averaged away against the quiet ones. That turns out to be an overloaded availability zone (one of the isolated data centers a cloud region is made of), fixed by the end of the week. The instrumentation was the incident response.

Histograms for anything latency-shaped: request durations, query times, queue waits. Summaries only when you don't need aggregation, like a single process, or a number you'll never merge. And when someone proposes client-side percentile computation, ask how the numbers merge. If the answer is "average the p99s," walk away.

Section 5 — Following one request

Metrics tell you a spike happened. They don't tell you where 2.1 seconds went.

To learn that, you follow one slow request's journey: the gateway took 12 ms, the auth service 12 ms, the checkout service 400 ms, and 2.1 of those seconds were spent waiting on the payment provider. That journey is a trace, built from spans: one span per unit of work, each with a name, a start time, a duration, and the ID of its parent. A trace is a tree of spans, and the slowest branch is your answer.

The mechanism is context propagation: the trace ID travels with the request from service to service, each one starting child spans under the same trace. On HTTP it rides in the W3C traceparent header (a W3C standard since 2020, with a companion tracestate header and a flag saying whether this trace is being sampled); on gRPC in the metadata; and, the part teams forget, on messages: when a request drops work onto a queue, the producer has to write the trace context into the message's headers and the consumer has to read it back, or the trace ends at the queue. For batch consumers that handle many messages at once, OpenTelemetry has span links so one consumer span can point at the many producer spans it's processing. Cron jobs start new traces linked to whatever triggered them. If the ID doesn't propagate, the trace breaks into orphaned fragments. This is an engineering discipline, not a library feature: every client wrapper, every queue producer, every async handoff must forward the context. The teams that struggle with tracing almost never struggle with the tracing system. They struggle with the three internal HTTP clients nobody instrumented, which is where the kernel-level instrumentation from Section 2 earns its keep.

Three IDs get called "the request ID," and they're different. The trace ID is the one the tracing system generates and propagates. A request ID is often something the load balancer stamps on the incoming request (post #11) and returns in a response header so a customer can quote it in a support ticket. A correlation ID is a business key, like an order number, that ties together several traces over hours. Put all of them in the logs. Only the trace ID needs to propagate hop by hop.

Then the expensive question: which requests do you trace? You can't keep all of them; at high traffic, full tracing is a storage and cost firehose. The two basic strategies:

• Head-based sampling: decide at the start of the request; keep 1 in 100, say. Cheap, simple, predictable cost. The flaw: the decision is random, so the interesting requests (the slow ones, the errors) are sampled at the same rate as the boring ones. And the rates compound: at 1% sampling and a 0.1% error rate, you keep one error trace in every hundred thousand requests, which is not enough to debug anything.
• Tail-based sampling: collect all spans briefly, decide at the end: keep the slow ones, the errors, a baseline of the normal ones, drop the rest. You keep nearly all the interesting traces at a fraction of the cost. The price is infrastructure rather than application code: services export every span as usual, and it's the collector tier (OpenTelemetry's tail-sampling processor) that buffers a trace's spans for a window, 30 seconds by default, before deciding. Since the decision needs every span of a trace in one place, all of a trace's spans have to be routed to the same collector instance, which means a load-balancing layer in front of the collectors that hashes by trace ID.

In practice, mature setups run several strategies at once: a low head-sampling rate as the baseline, rules that always keep errors and requests over a latency threshold, per-tenant rules so a big customer's traces aren't lost in the noise, and a rate limit on total spans stored so a traffic spike can't turn into a bill spike. Start head-based, because 1–10% with good propagation beats no tracing at all and it's an afternoon's work. Move to tail-based when incidents start hurting; it directly serves the 3 AM question ("show me the slow ones"). What you must never do is sample at 0.1% head-based and declare victory. That's Section 2's story, pillars without answers.

The outage that sells most people on tracing looks like this: checkout p99 spikes to 8 seconds. Metrics show the spike; logs show timeouts; neither shows where. One tail-sampled trace of a slow checkout shows the tree: gateway, checkout, inventory, and inside the inventory call, a DNS lookup taking 6 seconds. A DNS lookup. The inventory service's resolver was misconfigured after a migration, and only the trace, the actual journey of one unlucky request, could have found it, because no metric is labeled "DNS was weird for this one request." (One caveat: the trace only shows the DNS step if the HTTP client library emits DNS and connect spans. Many auto-instrumentations show one opaque 6-second client span, which still tells you which call to look at, but not why.) Traces answer the questions metrics can't ask.

Trace every service boundary from day one, even head-sampled, even at 1%. Propagate context through everything, including queues and cron jobs. Watch the SDK's overhead and attribute limits, because a span with a 2 MB request body attached is a span nobody wanted. And put the trace ID in your logs; that's what turns three pillars into one investigation.

Section 6 — Logs that answer questions

Everyone logs. Almost nobody logs well.

The typical production log is a diary: 2026-09-28 03:07:12 INFO checkout complete for user 8471 in 234ms. Readable by a human, useless to a machine. At 3 AM you don't need prose; you need to ask "show me all checkouts over 2 seconds in the last hour, grouped by payment provider, for users in region eu-west" and get an answer in seconds. A diary can't do that. Structured logging can: every log line is a record with named fields, {"event":"checkout","user_id":8471,"duration_ms":234,"provider":"stripe","region":"eu-west","trace_id":"abc-123"}. Same information, but now it's queryable. The log line stops being a sentence and becomes a row in a database you can slice. (JSON is the common shape; logfmt, the key=value style, is the other, and OpenTelemetry has its own log record model that either can feed.)

Three fields earn their keep on nearly every line: the trace ID (joins logs to traces), the duration (so slowness is searchable without a trace), and the outcome (success or error code as a field, not buried in a message). Everything else is per-service judgment. The discipline is small and absolute: no string interpolation of values into messages. f"checkout failed for {user}" is a diary; {"event":"checkout_failed","user_id":user} is data.

Three things about volume, because logs are the telemetry that grows fastest:

• Levels mean something. DEBUG is off in production, always, and turned on for one service for one hour when you need it. INFO is the record of what happened. WARN and ERROR are what you search first. A service logging at DEBUG in production is paying to store its own thinking out loud.
• Sample the routine, keep the rare. Log one in a hundred successful requests and every failed one. The successes are for trends and the failures are for incidents, and neither needs the other's volume.
• Retention is tiered. Hot, searchable storage for the incident window (a few days to a week or two), cheap cold storage for the compliance period, and a deletion schedule beyond that. Logs you keep forever are a bill and a liability.

Now the trap: cardinality. Cardinality is the number of distinct values a field takes, and structured fields are dimensions you can group by. region has 5 values: fine. user_id has 40 million values. In a metrics system, or in a logging system that indexes labels the way a metrics system does (Loki is the common one), every distinct value becomes its own series, and forty million series is a system that costs more than the product. Search-style log stores (Elasticsearch, Datadog) handle high-cardinality fields well for filtering (find this one user's requests) and choke on grouping (top users by error count across a day), so the rule is the same in both: high-cardinality values are fields you filter on, never dimensions you group by. For metrics, normalize: label with the route template /users/{id}, not the concrete URL /users/8471, or you get one series per user. For logs, keep both, the template as a groupable dimension and the concrete URL as a filter-only field, because at 3 AM "which ID?" is the question.

What never goes in a log. Passwords, session tokens, API keys, card numbers, and anything that would be a breach if the log system were compromised, which it will be, because log systems are copied into tickets, dashboards, and chat. Redact at the source, not in the pipeline. Personal data is its own category: a user_id in a log is a record you may be legally required to delete on request, and "we can't delete individual lines from cold storage" is the sentence you don't want to say to a regulator. Log what the investigation needs, and know your retention.

The version of the cardinality lesson most teams learn the expensive way: user_id gets added as an indexed dimension "for debugging," the logging bill triples in a month, and queries time out. The fix is a cardinality budget: an allowlist of groupable dimensions, everything else filter-only. The logs get cheaper and faster to query. Every field you index is a check you're writing against your future budget. Index deliberately. And once your logs are structured, notice that a lot of "log analysis" is really metrics: a counter of checkout_failed events by provider is cheaper to keep as a metric derived from the log stream than to grep out of the logs every time.

Section 7 — What to measure: RED, USE, and the golden signals

Engineers love measuring things. The failure mode is measuring everything: 400 graphs, none of which anyone can interpret at 3 AM. Three methods cut through the noise, and they're variations on one idea.

Google's SRE book named the four golden signals for any service: latency, traffic, errors, and saturation. Two later methods split those by what you're looking at.

RED, for services (Tom Wilkie's 2015 formulation, essentially the golden signals minus saturation):

• Rate: requests per second. Is traffic normal?
• Errors: failed requests per second, or the error ratio. Is it working?
• Duration: the latency distribution, p50 and p99. Is it fast?

USE, for resources (Brendan Gregg's 2012 method, for every thing your services run on: CPU, disk, network, database connections, thread pools):

• Utilization: how busy is it, as a percentage?
• Saturation: how much queued work is waiting, the part utilization hides?
• Errors: is the resource itself failing?

The RED dashboard is the 3 AM starting point for every service: rate, errors, duration, each broken down by endpoint. When the page fires, you open the RED dashboard for the suspect service and within a minute you know which of the three is wrong, and which endpoint. Rate tells you if it's a traffic problem, errors tell you if it's a correctness problem, duration tells you if it's a latency problem. If your service dashboard doesn't show RED, it's decoration.

USE matters because utilization lies by omission. A disk at 60% utilization looks fine until you see the saturation: 2,000 operations queued behind it. CPU at 70% with a run queue of 40 (forty threads waiting for a turn on the CPU) is not "30% headroom"; it's a traffic jam. The resilience post's (#2) cascading failures almost always show up in saturation before they show up in errors: queues growing, connection pools exhausted, thread pools full. Saturation is the early-warning radar. Utilization is the rearview mirror.

Then there's the question of what "normal" is. Traffic has a daily and weekly shape, and a graph of raw requests per second can't tell a Tuesday-morning dip from an outage. Two cheap answers: overlay last week's line on this week's, so the eye does the comparison; and for alerting, compare to a smoothed baseline (an exponentially weighted moving average, which weights recent minutes more than old ones) rather than to a fixed threshold. Seasonality-aware detection is a deep field; the overlay gets you most of the value.

Now the wallpaper problem: dashboards nobody looks at. A dashboard is a tool for a specific person at a specific moment, so design it that way. Top-down: the business number first (checkout success rate), then the service RED panels, then the resource USE panels, so a reader can go from "something's wrong" to "here" in three scrolls. One dashboard per audience: the executive view, the service owner's view, the on-call's view, each fitting on one screen. And every panel should drill down: click the p99 spike and land on the traces from that bucket (Section 2's exemplars), click the error rate and land on the matching logs. Then the audit every team should run yearly: for each dashboard, name the incident it helped debug. No incident? It's wallpaper. Delete it. The dashboards that survive are few: one RED per service, one USE per resource class, a single top-level "is the business healthy" view, and one that watches the monitoring itself (is the collector up, is the pipeline lagging), because the day the telemetry breaks looks exactly like the day everything is fine.

The 200-dashboard company exists everywhere. When a Sev-1 hits (a severity-one incident, the kind that pages leadership), nobody can find the right dashboard; engineers screen-share and scroll while the incident burns. The postmortem's fix is radical deletion: 12 dashboards survive. The next incident's time to diagnosis drops from 40 minutes to 6. They needed less visibility, not more, but findable.

RED on every service from day one: three panels, no excuse. USE on every resource you pay for. And if a dashboard hasn't earned its keep in an incident, it goes.

Section 8 — Alerts worth waking up for

Eighty alert rules. Phones buzz all night: disk 85% full on a dev box, CPU spike on a batch worker, a certificate expiring in 29 days, "error rate elevated" on an endpoint nobody owns. Engineers start sleeping through pages. Then the real one comes, checkout down, buried under eleven noise pages, acknowledged late. The postmortem's bitterest line: "we were paged 47 times that week and 46 of them didn't need a human at 3 AM." Pager fatigue is an alert-design problem, not a discipline problem. Every false page trains the team to ignore the true one.

The rule that fixes the targeting comes from Rob Ewaschuk's "My Philosophy on Alerting," which became the monitoring chapter of Google's SRE book: alert on symptoms, not causes. Users don't experience "disk 85% full." They experience "checkout is slow" and "the site is down." So page a human when a user-visible symptom is bad: p99 latency over threshold, error rate over threshold, checkout success rate dropping. Disk-full, CPU-high, queue-growing are causes. They belong in a ticket, a dashboard, or an automated remediation, not on a phone at 3 AM. Ewaschuk's test for a page is four words: it has to be urgent, important, actionable, and real. And a page should require intelligence; if the response is always the same command, that's a script, not a human. (He also allows the exception: a few cause-based alerts, like "disk full in four hours," are worth paging on because the symptom, when it arrives, will be much worse. Keep those rare.)

Then the framework that makes "symptom" precise: SLIs, SLOs, and error budgets, from the SRE book. An SLI (service-level indicator) is the measurement: the fraction of checkouts that succeeded within 500 ms, over the last five minutes. An SLO (objective) is the target for that indicator over a window: 99.9% of checkouts succeed within 500 ms, measured over 30 days. (An SLA is the contractual version with penalties; you set SLOs tighter than your SLAs so you notice first.) The error budget is what the SLO permits you to fail: 0.1% of 30 days is about 43 minutes a month of brokenness you're allowed. Spend the budget slowly and you investigate calmly. Burn it fast and you page someone. The error budget turns alerting from a feeling into arithmetic: you're not paging because "errors look high," you're paging because "we're consuming the month's allowed failures in hours."

Burn-rate alerting is the implementation. The burn rate is how fast you're spending the budget relative to the pace that would exactly exhaust it at the end of the window: burn rate = (observed error rate) ÷ (1 − SLO). At a 99.9% SLO, a 0.1% error rate is a burn rate of 1 (you'll use exactly your budget by day 30), and a 1.44% error rate is a burn rate of 14.4, which spends 2% of the month's budget in one hour. The SRE Workbook's recommended setup uses three tiers, each checked over two windows (a long one for accuracy and a short one, a twelfth as long, so the alert stops firing quickly once things recover):

Burn rate answers the question raw thresholds can't: "is this bad relative to what we promised?" The example that shows why it matters: a sustained 1% error rate against a 99.9% SLO is a burn rate of 10, an emergency. The same 1% against a 99% SLO is a burn rate of exactly 1: no page, but you'll have spent the entire month's budget by day 30, which is the "ticket and investigate this week" tier. And 0.1% against a 99% SLO is a tenth of the budget's pace, a quiet Tuesday. Thresholds don't know the difference. Burn rates do.

And the unglamorous one, non-negotiable: every paging alert gets a runbook. A short doc linked from the alert: what this alert means, the first three things to check, who owns the service, when to escalate. At 3 AM, nobody is clever. The runbook is the cleverness, written down at 3 PM by someone fully awake. An alert without a runbook is a fire alarm with no exit signs. And the corollary: if an alert fires and the runbook's first step is "check whether this matters," it doesn't. Delete it.

Four practices keep an alerting system trustworthy after it's built:

• Review the alerts, not just the incidents. Ewaschuk's rule is that an alert that's wrong more than half the time is broken, and even a 10% false-positive rate deserves a hard look. A weekly fifteen-minute look at "what paged, and did it need to" is where the 80-rule list gets pruned before it grows back.
• Test the alerts. Alert rules are code: unit-test them (Prometheus ships promtool test rules), fire the condition synthetically in staging, and add the two alerts that catch a dead monitoring system: an absent-metric alert for a service that stops reporting, and a dead man's switch, an alert that fires constantly and whose absence means the pipeline is broken.
• Know the pipeline's own latency. Scrape interval plus ingestion lag plus the evaluation window means a "one-minute" alert can't fire in one minute, and the observability system can itself be down during the outage it's meant to catch. Section 7's "monitor the monitors" dashboard is where that shows.
• Treat on-call as a system. Rotations with enough people that nobody is on call more than a week in four; a handoff ritual; a paging load that's tracked and treated as a bug when it climbs; and blameless postmortems (the SRE book's chapter is the model) for every real incident, feeding both the runbooks and the alert review. Time to detection, time to acknowledge, and time to recover (MTTD, MTTA, MTTR) are the numbers those postmortems track, and they're the same numbers the game days in Section 9 measure on purpose.

The team that replaces 80 threshold alerts with six burn-rate alerts on SLOs, each with a runbook, sees pages drop from 47 a week to 3. The on-call rotation goes from dreaded to boring, and the three pages that do fire are real, actionable, and resolved with the runbook. They got fewer interruptions, not fewer incidents. The pager should be boring 99% of the time and correct 100% of the time.

Section 9 — Going deep: the principal-level toolkit

Your checkout calls 10 services. Each one is excellent: p99 of 100 ms. What p99 does the customer see?

It depends on how the calls are arranged, and the two arrangements behave differently enough that it's worth being precise, because the sloppy version of this argument gets repeated a lot (including in the first draft of this post).

If the 10 calls are made in parallel and the request waits for all of them, the customer is slow whenever any one of them is slow. The chance that at least one of ten independent hops hits its 1% tail is 1 − 0.99¹⁰, about 10%. So ten parallel hops each at p99 give the customer roughly a p90 experience: one request in ten is slow, not one in a hundred. With a 100-way fan-out (a search that queries 100 shards and waits for all of them), the chance that at least one shard is in its tail is 1 − 0.99¹⁰⁰, about 63%, and the per-shard p99 is now something most customers experience on most requests. This is the central observation of Dean and Barroso's "The Tail at Scale," and it's why fan-out systems hedge requests (send a duplicate to a second replica when the first is slow, and take whichever answers first; the resilience post, #2, covers it), and why the sharding post (#3) is nervous about scatter-gather queries.

If the 10 calls are made in sequence, the latencies add rather than race. The customer's total is the sum of ten durations, so the typical experience is ten times one hop's median, and a slow hop adds its excess to the total rather than replacing it. There's still about a 10% chance that some hop is in its tail, and when it is, the total is worse than in the parallel case, because you pay the slow hop plus the nine others. So: deep sequential chains are slow all the time and have a bad tail; wide parallel fan-outs are fast when everything's fine and hit the tail more often than any single component would. The precise way to say the general rule is that the tail compounds across hops: the chance of avoiding every tail is 0.99ⁿ, which shrinks with every hop you add.

The design consequences are real: keep call graphs shallow (every hop donates its tail), set per-hop timeouts with budgets (the total budget is fixed, so each hop gets a slice, and the estimation post, #13, has the arithmetic for why per-hop p99s don't simply add up to an end-to-end p99), parallelize independent calls (parallel hops race, sequential hops accumulate), and hedge the fan-outs. When someone proposes adding "just one more service" to the request path, the principal asks what it does to the end-to-end p99. Usually nobody computed it. Now you will.

Observability cost control: the bill is real. Full-fidelity telemetry at scale is one of the largest line items in infrastructure budgets; Coinbase's roughly $65 million bill with one vendor (a year's spend, paid up front) made the news in 2023, and plenty of companies quietly find their observability spend within sight of their compute spend. The bill has three different shapes, and each responds to a different lever. Metrics cost is proportional to active series times retention, so the lever is cardinality. Logs cost is proportional to bytes ingested and indexed, so the levers are levels, sampling, and what you index. Traces cost is proportional to spans stored, so the lever is sampling. The controls, in order of leverage:

1. Sample aggressively, but intelligently. Tail-based sampling (Section 5) keeps the interesting traces and drops the boring ones; vendors and teams report trace-storage reductions somewhere between one and two orders of magnitude, depending on how boring the traffic was to begin with.
2. Retention tiers. Raw, high-cardinality data for days (the 3 AM window), downsampled aggregates for months (trends, capacity planning), cold archive beyond that. Nobody needs per-second granularity from last March.
3. Drop what you'll never query. Debug logs in production, health-check endpoints in traces, verbose spans from chatty libraries. Every byte stored should have passed a cost-benefit test.
4. Enforce it in the pipeline, not in a policy doc. Prometheus has a per-target sample_limit; relabeling rules can drop series before they're stored; the OpenTelemetry Collector can rate-limit spans per tenant and will apply backpressure (and drop, with a counter you should alert on) when a backend can't keep up.

Cardinality as a budget. Section 6 warned about the explosion; the principal version is organizational. The number of series a metric produces is the product of its labels' value counts: endpoint × region × version with 50 endpoints, 10 regions, and 10 live versions is 5,000 series for one metric, and the moment someone adds customer_id it's 5,000 times however many customers you have. So give every team a cardinality budget, a cap on distinct series and indexed log dimensions, and make exceeding it a review rather than a surprise bill. The budget forces the design question up front: is endpoint × region × version worth it, or does endpoint × region answer the same questions? Almost always the latter. Cardinality is the one observability cost that multiplies rather than adds, so it needs a governor, not just a graph.

Game days: closing the loop. The resilience post (#2) introduced game days, deliberately breaking things to practice recovery. Observability is what makes them useful: a game day without good telemetry is just an outage you scheduled. The mature loop: inject a failure (kill a dependency, add latency), then measure time to detection (how fast did the alert fire?) and time to diagnosis (how fast did someone find the cause using the tooling?). If detection took 20 minutes, your alerting is the bug. If diagnosis took an hour, your observability is the bug. Game days turn "we think we can see problems" into a measured number, and that number is the only grade that means anything your observability system gets.

A first observability game day tends to go like this: inject 500 ms of latency into one internal service at 2 PM on a Tuesday. Detection: 18 minutes, because the burn-rate alert's window was misconfigured. Diagnosis: 55 minutes, because traces weren't propagated through the async worker, so the trail went cold at the queue. Two gaps, both invisible until measured, both fixed in a week. The next game day: detection in 90 seconds, diagnosis in 6 minutes. Nobody bought better tooling. They measured the thing that mattered and fixed what the measurement showed.

The final reframe. Observability is usually sold as dashboards and vendors. It's bigger than that: the feedback loop that makes every other discipline work. Resilience without detection is hope. Canary deploys (one instance on the new version first; the load balancer post, #11) without measurement are gambling. Capacity planning (the estimation post, #13) without history is guessing. The eight concepts before this one were about building systems that survive scale; this one is about seeing them, because a system you can't interrogate is a system you don't operate. Build the system. Then build the eyes.

What Broke at 3 AM, distilled

For the skimmers and the revisitors: everything above, on one page.

Key numbers and rules:

Every trade-off, in one table:

Three ideas to take with you:

1. Monitoring answers the questions you predicted; observability answers the ones you didn't. The 3 AM test is the discipline: a new question about production, answered in under ten minutes. Everything in this post serves that test.
2. Averages lie, and tails compound. Measure latency in percentiles, compute them with histograms, and remember that ten parallel hops at p99 give your customer a p90. The tail is where your users live when things break.
3. Page on symptoms, ticket the causes, and prove it with game days. Multi-window burn-rate alerts on SLOs, a runbook on every page, and a measured detection time. The pager should be boring 99% of the time and correct 100% of the time.

Further reading

• Google SRE Book: Monitoring Distributed Systems. Chapter 6, by Rob Ewaschuk: the four golden signals and why symptoms beat causes.
• Rob Ewaschuk: My Philosophy on Alerting. The original essay behind that chapter; "urgent, important, actionable, real."
• Google SRE Workbook: Alerting on SLOs. The multi-window, multi-burn-rate design in Section 8, worked through; and Implementing SLOs for choosing the SLIs.
• Prometheus: Histograms and summaries. Why quantiles don't aggregate, and how histogram_quantile does; behind Section 4.
• OpenTelemetry: Sampling and Tail sampling with the Collector. Head versus tail sampling and where the buffering actually happens; behind Section 5.
• W3C Trace Context. The traceparent header; the propagation standard.
• Dean & Barroso: The Tail at Scale (CACM 2013). The fan-out arithmetic in Section 9 and the case for hedged requests.
• Brendan Gregg: The USE Method. Utilization, saturation, errors for every resource; behind Section 7.
• Grafana: The RED Method. Tom Wilkie's rate, errors, duration for every service; behind Section 7.
• Charity Majors, Liz Fong-Jones, George Miranda: Observability Engineering (O'Reilly; the 2026 second edition adds Austin Parker). The book-length argument for asking new questions of production; the deep end of Section 1.

Where you'll meet this

This post is the detection half of the resilience post's (#2) loop: retries, circuit breakers, and failovers are blind without the signals from this post. An alert is what starts the resilience machinery, and a game day is what proves both halves work. It leans on the replication post (#5): "is the replica behind?" is a monitoring question, with replication lag as a first-class metric earning its own burn-rate alert, and that post's Section 4 question, "did my write land?", is unanswerable without the request-scoped tracing from Section 5. The load balancer post (#11) is where the request ID gets stamped and where canary deploys wait for these metrics before rolling forward. The estimation post (#13) has the arithmetic behind the latency budgets in Section 9. And in the URL-shortener design, every step assumed invisible instrumentation: the redirect path's p99, the cache hit ratio, the write lag to replicas. This post is the dashboard wall that design deserved. Next up is async processing (#10), the part of the system where traces most often go cold.

Keep exploring: the Core Concepts series

Every post in the series stands alone. Read them in any order.

• From Laptop to Billions of Clicks: Designing a URL Shortener in 11 Steps — the anchor: one design, every concept under load.
• #1 The 100:1 Superpower: Caching — the fastest request is the one you never make.
• #2 The Blast Radius: Surviving the Day Your Dependencies Fail — staying up when everything you depend on goes down.
• #3 Divide and Conquer: Sharding and Partitioning — splitting one database into many without losing your mind.
• #4 The Snowflake Problem: Unique IDs at Scale — naming things when millions are born every second.
• #5 Copies of the Truth: Replication — keeping copies of your data that actually agree.
• #6 Do No Harm Twice: Idempotency — making "just retry it" safe.
• #7 The Copy at the Doorstep: CDNs and Edge Computing — serving from next door instead of across the ocean.
• #8 The Bouncer's Math: Rate Limiting — saying no politely, at scale.
• #9 What Broke at 3 AM: Observability, p99, and Useful Alerts — knowing what's wrong before your users tell you. (this post)
• #10 Do It Later, On Purpose: Async Processing and Queues — the work the user doesn't have to wait for.
• #11 The Traffic Cop: Load Balancing — the box in every diagram nobody explains.
• #12 Assume They're Already Knocking: Security and Abuse at Scale — designing for the users who are designing against you.
• #13 Do the Math First: Estimation for System Design — the two minutes of arithmetic that choose the architecture.
• #14 The Whiteboard Playbook: Taking On Any System Design Challenge — the whole series in forty-five minutes.

Blueprints of Scale — Core Concepts #9. If this helped, the best thanks is a share with someone who's learning.

That gap is this post. Performance asks what happens when a million of the users you built for show up. Security asks what happens when one user you didn't build for shows up with a million requests. They're different questions with different designs, and the second one is the one that wakes you up at night, because ordinary users forgive a slow Tuesday and the dishonest one never stops probing.

Here's what's covered: how to think like a defender (threat modeling without the jargon, and the question of what you could simply stop storing); why "unlisted" is not "private," how enumeration works, and the difference between hiding an ID and checking who's allowed to see it; signed URLs and the discipline around secrets, service identity, and encryption keys; input validation, the actual state of injection, CSRF, and the browser-side headers; the spam and malware screening pipeline the interview pointed at this post; credential stuffing and account takeover, with password hashing, sessions, and the kind of MFA that actually resists phishing; bot mitigation and why it isn't rate limiting; DDoS tiers and what the edge absorbs; detection, honeypots, audit logs, and alerting on abuse; and the principal-level playbook: defense in depth, abuse economics, the first hour of a real incident, and the security-versus-usability trade nobody lets you dodge.

If "threat model" sounds like a job title, start at Section 1; the first two sections assume nothing, and every term is defined the first time it appears. Sections 3 through 8 are the machinery you'll build and run. Section 9 is the judgment. There's a one-page cheat sheet at the end under Assume They're Already Knocking, distilled, and every diagram is also described in the text around it.

Section 1 — You have two user bases

Every system has two user bases. The first is the one you designed for: people shortening links, uploading photos, buying things. The second is the one designing against you. They're more creative than your product team, better funded than your roadmap, and they never file bug reports. They take what they want and leave.

The way this usually gets learned is by accident. Picture a free image-resize API launched as a developer-relations play: generous limits, no authentication, meant to be a billboard. Within weeks someone notices the resize endpoint will fetch any URL you hand it and return the bytes. Now it's a free anonymous proxy for scraping competitors, a free host for phishing pages (which carry the company's domain, lending them credibility), and a very expensive billboard once the bandwidth bill arrives. The team hadn't built a product. They'd built infrastructure for strangers, at their own expense, because nobody had asked, at any point, how someone would abuse this.

That question is threat modeling, and for a backend engineer it's simpler than the name suggests. Three questions, in order:

1. What do we have that's worth taking? Data (user records, private content, credentials), compute (CPU, GPU, bandwidth, all of which cost real money per hour), and trust (your domain's reputation, your email deliverability, your brand on someone else's phishing page). Attackers are rational. They want the asset with the best return for the least effort. And there's a second half to this question that gets skipped: what could we stop having? Every field you don't store can't be stolen. Data minimization (collect what the feature needs, keep it only as long as the feature needs it, delete the rest on a schedule) is the cheapest security control there is, because the scope of a breach is exactly the data you kept.

2. Who wants it, and what can they do? The cast is small and worth memorizing. Scrapers want your data in bulk. Spammers want your reputation and your users' inboxes. Fraudsters want money moving through your system, and they'll use your own features to move it: coupon farming, referral abuse, refund fraud, chargebacks. Miners want your compute. Extortionists want your uptime (pay, or the flood continues). And the bored and curious want to see if the door is unlocked. Different attackers need different defenses; you don't fight a scraper and a DDoS with the same tool.

3. Where are the doors? Every input is a door: every API endpoint, every form field, every file upload, every webhook (an HTTP request another service sends you when something happens on their side), every URL parameter, every header your code reads. Your attack surface is the complete list of places where the outside world touches your system. You can't defend a door you don't know exists, which is why the first security exercise for any system is just listing them.

Read it left to right: five kinds of attacker, one system, three kinds of loot. Every section from here on is a door with a lock designed for a specific kind of knocking.

Security is not a feature you add but the assumption you design under: someone is already knocking.

Section 2 — "Unlisted" is not "private"

Back to the interviewer's question. The URL shortener minted keys from a counter (1, 2, 3) encoded in Base62 (digits plus both cases of letters, 62 symbols). Key 125 becomes cb, key 126 becomes cc. Anyone who sees one link can guess the next. So an attacker writes a loop: fetch /abc001, /abc002, /abc003. A hundred thousand requests later they hold every link ever created, including the "private" ones: the unlisted document share, the surprise-party invitation, the confidential draft. Nothing was hacked. No password was cracked. The attacker counted.

This is an enumeration attack: guessing identifiers in sequence. It shows up wherever identifiers are sequential: auto-incrementing order numbers, timestamps in filenames, dates in paths. The root cause is always the same. The identifier is predictable, and predictability is a kind of publicity. "Unlisted," meaning not linked from anywhere, feels private. It isn't. It means the attacker has to count instead of click.

Before the fixes, one distinction that the shortener example blurs, because it matters more than anything else in this section. There are two different situations:

• Nobody is logged in, and the link itself is the permission. A share link, a password-reset link, an invite code. Here the ID has to be unguessable, because there's nothing else standing between the attacker and the resource.
• Someone is logged in, and they're asking for an object. /orders/10492. Here the ID being guessable is not the bug. The bug is that the server hands over order 10492 without checking whether the logged-in user owns order 10492. This is IDOR (insecure direct object reference; the API-security world calls it BOLA, broken object-level authorization), and it has been the number one item on OWASP's API Security Top 10 since the list existed. The fix is not a random ID but an ownership check on every object, every function, and every property, done in one central place rather than remembered per endpoint, and tested by logging in as two different users and trying to read each other's things.

The words for those two things are authentication (who are you) and authorization (what are you allowed to do). Most real-world "enumeration" breaches are authorization failures that get told as enumeration stories: the IDs were guessable and nobody checked. Random IDs help with the first half. Nothing but an authorization check helps with the second.

Now the fixes for the first situation, where the link is the permission.

Unguessable IDs. 128 bits of randomness (a UUIDv4 has 122 random bits). The arithmetic on collisions is worth doing once, because "random" makes people nervous: with 122 random bits you'd have to generate about 2.7 quintillion IDs (2.7 × 10¹⁸, a billion a second for 86 years) before the odds of any two colliding reach 50%. At a billion IDs the probability is around one in ten billion billion. A random ID can't be enumerated because there's no sequence to follow: after seeing 9f3a…, the attacker's next guess is as good as their first, and "as good as the first" at 122 bits means never. Use this for anything that must stay private without authentication: share links, password-reset tokens, invite codes. Two caveats. A random ID in a URL still leaks the ordinary ways URLs leak (browser history, Referer headers, server logs, link-preview bots that fetch anything pasted into a chat), so treat it like a bearer token and expire it when you can. And randomness is one of two ways to make a link its own permission; the other is a signature, which is Section 3.

Break the visible sequence, correctly. Sometimes you need short IDs that stay sequential underneath (databases love them, and seven characters is the whole point of a shortener). The instinct is to scramble the counter before encoding it, and the two scrambles people reach for first are broken in a way that's worth understanding. XOR-ing the counter with a fixed secret (XOR is the bitwise operation where each bit of the output is 1 if the two input bits differ) preserves the differences between inputs: 125 and 126 differ only in their lowest bits, so 125 ⊕ K and 126 ⊕ K also differ only in their lowest bits, and after Base62 encoding the two "scrambled" keys share every character but the last. Enumeration becomes "increment the last character." Shuffling the Base62 alphabet is even weaker; it's a substitution cipher on digits, and a handful of known key pairs reveals the whole mapping. What you want is a keyed permutation that scrambles all the bits together: run the counter through a small Feistel network (a few rounds of keyed mixing, reversible, sized to your key space), or use a format-preserving encryption mode like NIST's FF1. Beware of the libraries that only look like this: Sqids, for example, is an alphabet shuffle with a per-ID offset underneath, and its own documentation says it plainly, no encryption of any kind, so it makes IDs pretty without hiding the sequence from anyone who collects a few. A real permutation stops casual counting. It does not stop a determined reverse-engineer, and it's never a substitute for randomness where privacy matters.

Rate-limit the misses. Enumeration against unguessable or scrambled IDs generates floods of 404s, because most guesses miss. A per-client limit on misses (the rate limiting post's machinery, #8, pointed at a different signal) turns "scrape everything tonight" into "scrape everything over the next few months." It doesn't fix predictability; it makes exploiting it boring.

Don't confirm existence. When an unauthenticated request hits an ID it shouldn't see, return 404, not 403. A 403 says "this exists and you can't have it," which tells the enumerator to keep that ID on the list. A 404 says "nothing here." Two refinements: this rule is for unauthenticated and cross-tenant requests, and a logged-in user who lacks permission for something they can legitimately know exists is often better served by an honest 403. And the 404 has to be a convincing 404: same response size, same timing as a real miss, or the difference confirms existence just as well as the status code would.

The layered answer: unguessable IDs where the link is the permission (the real fix), a keyed permutation where IDs must stay short and sequential underneath (the cheap fix), miss-rate limiting to raise the cost of trying (the economic fix), 404-not-403 so failures leak nothing (the hygiene fix), and an authorization check on every object regardless (the fix for the other half of the problem). No single one is sufficient. Together they're annoying enough that the attacker goes elsewhere, and Section 9 will explain why "go elsewhere" is a legitimate security strategy.

Section 3 — Sign it or lose it

Some links have to work without a login: the password-reset email, the private file download, the "your invoice is ready" link. You can't put a login screen in an email. So you put a signature in the URL instead: a cryptographic stamp that says "whoever holds this link was given it by us, for this resource, and it expires soon." That's a signed URL, and it's worth learning precisely because the construction is simple.

The recipe: take the resource (/invoices/9042.pdf), add an expiry timestamp (expires=1790535802), and compute an HMAC over that string with a secret key only your servers know. (HMAC, hash-based message authentication code, is a standard way to turn a hash function like SHA-256 plus a secret into a tamper-proof stamp; RFC 2104 specifies it in eleven pages.) Append the result: /invoices/9042.pdf?expires=…&sig=9f3a…. When the request arrives, the server recomputes the HMAC over the resource-plus-expiry and compares it with the sig in the URL, using a constant-time comparison so that how long the comparison takes doesn't leak how many bytes matched. Match: serve it. Mismatch or expired: reject. The attacker can see the URL format, the expiry, and the signature, and still can't mint a valid link for a different file, because minting requires the secret.

What the signature must cover is where implementations go wrong: the exact resource (path, plus any query parameters that change what's served), the expiry (a signed URL without an expiry is a permanent credential; a password-reset link that works forever is a second password), and the action if the URL grants one (download and delete must not share a signature). Anything the server acts on but the signature doesn't cover is a forgery waiting to happen. Sign file=9042&action=view but ignore action during verification, and the attacker flips it to action=delete with a still-valid signature.

Once you see this pattern you see it everywhere. S3 presigned URLs are the same idea (AWS's version derives the signing key through a chain of dates and regions, but it's HMAC-SHA256 with an expiry underneath). Password-reset links. Email-unsubscribe links. And webhook verification, which is the version every backend engineer eventually implements: when Stripe sends your server a webhook, it includes a Stripe-Signature header containing a timestamp and an HMAC-SHA256 over "{timestamp}.{raw body}" computed with a secret only you and Stripe share. You recompute, compare in constant time, and reject anything older than five minutes so a captured webhook can't be replayed later. Any time your server acts on an unauthenticated request, the request should carry proof it came from you or from someone you trust. The signature is that proof.

Signed URLs are only as safe as the secret, and secrets are where backend teams actually bleed. The discipline:

• Secrets live in a vault or the platform's secret store, never in code, URLs, or logs. Every secret that has been committed to git is compromised; rotate it and move on. Log lines get copied into tickets, dashboards, and chat, so a secret that passes through a log formatter will eventually be read by someone who shouldn't have it.
• One secret per purpose. The webhook-signing secret is not the session-encryption secret is not the API key. When one leaks (when, not if) the blast radius is that one purpose.
• Rotate without downtime. Every secret gets a version or key ID; verification accepts the current and the previous secret during rotation; signing uses only the current one. Stripe does exactly this, letting you keep an old webhook secret valid for up to 24 hours after you roll it. Rotation is then a deploy, not an incident. A secret you can't rotate is a secret you'll keep after it has leaked.
• Minimal scope and short life. A key that can only read one bucket is a smaller disaster than a key that can do everything. Signed URLs expire in minutes or hours, not months.

The same logic extends from secrets to identities. Every service should have its own identity with only the permissions it needs, and its credentials should be short-lived and issued by the platform rather than pasted from a spreadsheet: IAM roles on AWS, workload identity on GCP and Kubernetes, and inside a mesh, per-service certificates (post #11 shows where mTLS sits in a service mesh). This matters concretely in Section 4, where the prize for a server-side request forgery is often the cloud metadata endpoint that hands out a machine's credentials; on AWS, requiring IMDSv2 (which needs a session token a simple forged GET can't obtain) closes the easy version of that. And the certificate lifecycle is its own outage class: automate renewal (ACME) and alert on expiry.

One more thing that lives in the same drawer: encryption at rest, because "the disk is encrypted" is the sentence that makes people feel safe and shouldn't. Full-disk encryption protects against someone walking off with the drive. It does nothing against an attacker who is talking to your application, because the application decrypts everything for them. The layer that helps against application-level attackers is field-level: encrypt the crown-jewel columns (card numbers, government IDs, health data) with keys the application fetches from a key management service (KMS) at runtime. The standard construction is envelope encryption: a data key encrypts the row, a master key in the KMS encrypts the data key, and the master key never leaves the KMS, so rotating it means re-wrapping data keys rather than re-encrypting terabytes. For payment cards, most teams don't store the number at all; they store a token from the payment provider, which shrinks the compliance scope to almost nothing. That's the second half of Section 1's first question again: the safest column is the one you don't have.

The story worth carrying around from this section: a team stores their payment-webhook secret in an environment variable, which is fine, and then a debugging endpoint dumps the process environment as JSON "temporarily," for three months. The secret isn't stolen so much as published, to anyone who finds the endpoint. Nothing bad happens, which is the worst outcome, because a secret can be compromised for months with zero symptoms. They rotate it, delete the endpoint, and add two rules: no endpoint ever returns environment contents, and secrets get a quarterly rotation whether they need it or not. Paranoia on a schedule is just hygiene.

Section 4 — Never trust a byte

Every byte that crosses into your system from the outside is hostile until proven otherwise. The username, the file upload, the webhook payload, the HTTP header, the query parameter, the JSON body with the helpful field named __proto__. Your code's job at the boundary is interrogation: is this the right shape, the right type, the right size, from someone allowed to send it? Everything downstream gets to assume the answer was yes. Validation is the one place in your architecture where paranoia is the job description.

The classics, and the state of each:

SQL injection is, in practice, a solved problem, and it's still on the OWASP Top 10 (A05 in the 2025 edition) because people keep un-solving it. Parameterized queries (prepared statements) separate code from data at the protocol level; the database cannot confuse a username for a command. Every ORM (the object-relational mapper your framework uses to turn objects into SQL) parameterizes under the hood. The holes are the places parameterization doesn't reach, and they're worth naming: you can't parameterize an identifier, so a table name, a column name, or an ORDER BY clause built from user input needs an allowlist instead; dynamic SQL inside stored procedures; and every ORM's raw-query escape hatch. If your code concatenates strings into SQL anywhere, that's where the vulnerability is.

Cross-site scripting (XSS) is an output problem. The fix isn't on input; it's on output. When you render user content into HTML, encode it so the browser treats it as text, not code. Modern frameworks do this by default; the holes are the escape hatches (React's dangerouslySetInnerHTML is named accurately) and anywhere you build HTML by hand. The backstop is a Content Security Policy header (CSP), which tells the browser which script sources are allowed at all, so an injection that slips past encoding still can't load the attacker's code.

Allowlists beat denylists. "Block <script>" is a losing game, because attackers have a thousand encodings you haven't thought of. "Only allow letters, numbers, and these five punctuation marks" is a winning one: the set of acceptable inputs is finite and yours. Validate type, length, range, and format at the boundary; reject what doesn't fit; never "clean" hostile input into something acceptable. Rejection is a security decision. Sanitization is a hope. Two traps inside "format": normalize Unicode before you compare (there are several byte sequences that display as the same character, and an attacker will register the one your uniqueness check doesn't see), and treat deserialization of anything but plain data formats as code execution, because for many languages' native serializers, it is.

File uploads are a special circle. Never trust the extension or the declared content type; read the actual bytes. Store uploads outside the web root, serve them from a separate origin (a different domain, so a malicious file can't run in the context of your site), and send them with an explicit Content-Type, Content-Disposition: attachment, and X-Content-Type-Options: nosniff, which together stop the classic trick of an SVG uploaded "as an image" that executes as script when someone opens it directly. Scan them (Section 5's pipeline), size-limit them, and never let the uploader choose the served filename or path; a filename containing ../ is a path traversal attack, an attempt to write or read outside the directory you intended.

SSRF: your server is the attacker now. Server-side request forgery is what happens when your code fetches a user-supplied URL (webhooks, link previews, image proxies, the resize API from Section 1) and the server makes the request with the server's network access. An attacker hands it a URL pointing at your internal network: the cloud metadata endpoint that hands out credentials, internal admin panels, services that trust anything from "localhost." The defense is an allowlist of destinations plus a hard block on private IP ranges and the metadata address (169.254.169.254), checked after DNS resolution and pinned, because DNS rebinding (the hostname resolving to a public IP at check time and a private one at fetch time) is how attackers get around a check done on the hostname. Two more that the checklists miss: don't follow redirects, because a redirect from an allowed host to the metadata IP walks straight past the pinned check; and require IMDSv2 or its equivalent so a forged GET can't fetch credentials even if it reaches the endpoint. Any feature that makes your server fetch a URL is a proxy; design it like one.

CSRF: the browser as a confused deputy. Cross-site request forgery is SSRF's client-side cousin. A user is logged into your site; they visit an attacker's page; that page contains a form that auto-submits a POST to your /transfer endpoint, and the browser helpfully attaches the user's cookies. The user made a request they never intended. The defenses are the SameSite cookie attribute (which stops the browser sending cookies on cross-site requests; Chromium-based browsers treat Lax as the default when it's missing, Firefox and Safari don't, so set it explicitly), anti-CSRF tokens on state-changing forms, and, for APIs, requiring a custom header that a cross-site form can't set. The legitimate exception: your webhook endpoints have no session and should accept cross-origin POSTs, which is why they get signatures instead (Section 3).

CORS is not access control. Cross-origin resource sharing headers tell browsers which other sites may read your responses from JavaScript. They do nothing to a curl command. A permissive Access-Control-Allow-Origin: * on an authenticated API is a bug, but a strict one isn't a defense against anything that isn't a browser. Server-side authorization (Section 2) is the defense.

The headers you set once. Strict-Transport-Security (HSTS) tells browsers to never talk to you over plain HTTP again, X-Content-Type-Options: nosniff stops content-type guessing, Content-Security-Policy is the XSS backstop above, and frame-ancestors (in CSP) stops your pages being embedded in someone else's frame for clickjacking. Each is a line of configuration. Set them, then verify them with any of the free header scanners.

Two more doors that aren't inputs in the obvious sense:

Logs. Logs are consumed by machines: a SIEM (security information and event management system, the thing that collects logs across the company and hunts for patterns), alerting pipelines, dashboards. A newline smuggled into a username field becomes a forged log line; a forged log line becomes a false alert or a hidden real one. Treat log fields like output: encode or strip control characters. And that __proto__ field from the first paragraph is prototype pollution, a JavaScript-specific injection where a crafted JSON key modifies the behavior of every object in the process; the fix is a merge routine that refuses __proto__, constructor, and prototype keys (or objects created without a prototype at all), not the parser.

Your dependencies. The 2025 OWASP list renamed and broadened its long-standing vulnerable-components category into Software Supply Chain Failures (A03), and it deserves a sentence here because it's the door that isn't an input at all. The code you npm install or pip install runs with all of your permissions. The controls are boring and effective: lockfiles committed, container images pinned by digest rather than tag, a software-composition scanner that tells you when a dependency has a known vulnerability, and CI credentials that can't publish to production. A compromised build pipeline doesn't need to find a door; it's already inside.

A word on the WAF, since Section 8 and the load balancer post both mention it. A web application firewall sits at the edge and pattern-matches requests against known attack signatures: SQL keywords in suspicious places, script tags, path-traversal sequences. It's useful for reducing noise and for "virtual patching" (blocking an exploit pattern in the hours before you can ship the real fix). It is not a substitute for anything in this section, because it's a denylist by nature and Section 4's whole point is that denylists lose.

The principal's version of this section: one input boundary per trust zone, and safe-by-construction interfaces inside it. Section 9 will argue for defense in depth, and this isn't a contradiction. Parameterized queries and encoded output aren't "re-validating"; they're interfaces that can't be misused. The tax is on redundant checking, not on building things that are safe regardless of what they're handed. In a service architecture every service is its own trust zone with its own boundary, and the interior of each one trusts its boundary.

Section 5 — The spam pipeline

Step 9 of the interview listed "spam/malware screening" and moved on: each of those is its own design problem. This is that design problem. A URL shortener is a phisher's best friend, because your domain's reputation launders their malicious link. A file host is a malware distributor's CDN. Anywhere users submit content, abuse follows, and the architecture that handles it has a shape worth learning, because it's the same shape everywhere.

The core tension: users expect instant, and abusers exploit instant. If every submission waits for human review, the product is dead. If everything publishes immediately, the abuse is live before you see it. The resolution is a pipeline: accept fast, judge slow, act at the speed the judgment warrants.

1. Hash check: the cheap instant verdict. Compute the content's hash and compare it against databases of known-bad content: malware signatures; phishing URLs; and for images, known child-abuse material (CSAM), matched with perceptual hashes like Microsoft's PhotoDNA that recognize visually similar images rather than byte-identical ones, so re-encoded abuse content still matches. A hit blocks instantly, no further thought. This is a constant-time lookup for exact hashes (perceptual matching is a nearest-neighbor search, a bit more work) and it catches the lazy majority of abuse, because most of it is recycled rather than novel. For URLs, the shared industry list is Google Safe Browsing, which Chrome, Firefox, Safari, and most other browsers consult (Edge uses Microsoft's SmartScreen instead). Note that the free Safe Browsing API is licensed for non-commercial use only; a product that makes money uses the paid Web Risk API.

2. Reputation and heuristics: the fast statistical verdict. New account? First submission? Link to a day-old domain? Mismatched geolocation? Each signal is weak; together they score. Below threshold: publish. Above: hold for review. The scoring runs in milliseconds, inline with the request.

3. Async deep analysis: the slow certain verdict. Sandboxed execution of attachments, classifiers on text and images, following links to their final destination. This runs in a queue (post #10), seconds to minutes after the user already got their "received" response.

4. Human review: the edge cases. The queue nobody wants to need and everybody needs: appeals, novel abuse, classifier uncertainty. Reviewers get the content with context (account history, similar past decisions) and their decisions feed back into step 2's models.

5. The feedback loop. Every confirmed bad submission becomes a hash for step 1 and a training example for step 2. The pipeline gets smarter with every attack it sees, which is the only sustainable advantage, because the attackers are iterating too.

Read it as two speeds: the left column decides in milliseconds (hash, reputation), the right column decides in minutes (deep analysis, humans), and the takedowns feed back into the instant checks. The economics: false positives cost you users (legitimate content blocked, appeals, anger), false negatives cost you everyone, because phishing from your domain poisons your reputation with browsers and email providers. Tune the thresholds like a business decision, because that's what they are: how much abuse you can tolerate versus how much friction your users can tolerate. There's no correct setting, only a correct setting for your tolerance.

Two things specific to shorteners and to email that the pipeline alone doesn't cover. A URL shortener is an open redirect by construction: it sends anyone anywhere, which is exactly the primitive phishers want and exactly why reputation systems are harsh on shorteners. The mitigations are a screening check at creation time plus periodic re-scanning (links rot into malware after they're created), an interstitial warning page for links to untrusted destinations, a kill switch that disables a link in seconds, and a report endpoint. And if your product sends email, the deliverability reputation Section 1 counted as an asset is protected by three DNS records, SPF, DKIM, and DMARC, which let receiving mail servers verify that mail claiming to be from your domain actually is; without them, a spammer can send from your domain and the resulting blocklisting is yours.

Here's what skipping step 1 looks like. A link shortener launches without screening ("we'll add it when we're bigger"). Within a month of the first phishing campaign, browsers begin flagging links on the domain as deceptive; Safe Browsing entries can cover a single URL, a path prefix, or a whole host, and heavily abused shorteners tend to end up with the whole host. Now every legitimate user's link shows a red warning page. Recovery takes quarters: delisting requests, appeals, rebuilding sender reputation. Abuse screening isn't a feature for when you're big. It's the thing that lets you get big without your domain becoming a warning label.

Section 6 — Your users are the target

Your authentication is only as strong as your users' worst password habit, and the habit is reuse. Billions of username-password pairs from a decade of breaches circulate in "combo lists," and the attack is brutally simple: credential stuffing replays those pairs against your login page, thousands per minute, until some of them work. Nobody is guessing passwords. They're replaying them. If 0.1% of attempts succeed, which is a realistic rate, a million-attempt run hands the attacker a thousand accounts: email access, saved payment methods, identity-theft starter kits.

Two things stop this before it starts.

Check new passwords against breach lists. Troy Hunt's Have I Been Pwned service exists for exactly this. The client hashes the password with SHA-1, sends only the first five characters of the hash, and receives back every breached-password hash suffix in that range (around 800 to 1,000 of them); the client checks for a match locally. The service never learns the password or even whether it matched; that's the k-anonymity design. "This password has appeared in breaches; pick another" stops reuse at the source, and NIST's password guidance now says you must do this.

Hash passwords so your breach doesn't become the next combo list. Store passwords with a deliberately slow, salted hash designed for the job: Argon2id, scrypt, or bcrypt, with a per-user salt and a cost factor tuned so one hash takes tens of milliseconds. Never SHA-256 or MD5, which a GPU computes billions of times per second, turning your leaked table into plaintext over a weekend. If you inherited fast hashes, re-hash each user's password at their next login.

But the attack is ongoing, so the real game is detection: recognizing a stuffing run while it's happening.

• Velocity anomalies. One IP attempting hundreds of logins against hundreds of usernames is not a forgetful user. One username hammered from fifty IPs is a distributed brute-force attack on that one account. And the sneakier shape, password spraying, is one common password (Summer2026!) tried once against thousands of usernames, which dodges per-account lockouts because no account sees more than one failure.
• Success-rate anomalies. Normal login traffic succeeds most of the time. A stuffing run fails 99.9% of the time, so a sudden crater in your login success rate is the signal, visible before any single account is confirmed breached. The caveat: the crater is only visible when the attack's volume rivals your legitimate traffic. A low-and-slow run hides in the aggregate, which is why you slice the rate per username, per IP, per network.
• Impossible travel. A login from New York followed ten minutes later by one from Singapore is either a VPN or a stolen session. This is a high-false-positive signal (corporate proxies and mobile carriers produce it constantly), so it's a trigger for stepping up authentication, not for blocking.
• The takeover choreography. New device, password change, email change, then a withdrawal or purchase, in that order, within the hour. Each step is legitimate-looking; the sequence is the tell. Account takeover has a dance, and a "log out everywhere" on password or email change breaks the dance mid-step.

Now the trap: account lockouts are a weapon. "Lock the account after five failures" sounds protective, until an attacker deliberately fails five logins on your account and locks you out. They've turned your defense into a denial of service they can aim at anyone. The mature answer is to throttle and challenge (slow the attempts, demand a CAPTCHA, require a second factor) rather than hard-lock, and to never reveal whether the username exists: "invalid username or password" for both cases, with identical timing, which means running a dummy password hash even when the user doesn't exist so the response takes the same tens of milliseconds either way. A login page that distinguishes "wrong user" from "wrong password" is an oracle for enumerating your user base, which is Section 2's lesson with a login form. The same oracle hides in signup ("that email is already registered") and password reset ("we've sent a link" versus "no account found"), so those flows get the same treatment.

Which key you rate-limit on matters here, and IP is the weakest one (and only as trustworthy as the X-Forwarded-For value your own load balancer appended, never the one the client sent; the load balancer post, #11, Section 3, has the rule). Behind a mobile carrier's NAT (address translation, which hides thousands of phones behind one public address) one address is thousands of users, and an attacker with a proxy pool has thousands of addresses. Limit per target username (so nobody can hammer one account from anywhere), per account after login, and per device or TLS fingerprint where you can; the rate limiting post (#8) has the mechanics.

Sessions and tokens are the second half of authentication, and a stolen session skips the password entirely. The rules are short. Session cookies get HttpOnly (JavaScript can't read them), Secure (HTTPS only), and SameSite. Rotate the session ID on login and on any privilege change. Keep the ability to revoke sessions server-side, and use it: a password change or an email change should invalidate every other session the account has. If you use JWTs (JSON Web Tokens, self-contained signed tokens the server doesn't have to look up), know their sharp edges: never accept alg: none, pin the algorithm, put nothing secret in the payload (it's only encoded, not encrypted), keep them short-lived because you can't revoke them, and pair them with a refresh token that rotates on every use and gets revoked if an old one is ever replayed.

And the backstop for all of it: multi-factor authentication (MFA), specifically the phishing-resistant kind. The distinction matters, because "MFA" covers things that are not equally strong. SMS codes are the weak form; SIM-swapping and SS7 attacks are routine for targeted victims, though SMS still defeats almost all automated stuffing. Authenticator-app codes (TOTP, time-based one-time passwords) are better, but they are not phishing-resistant: a real-time phishing kit that proxies your login page captures the code and uses it within its thirty-second window, and NIST says so explicitly. The phishing-resistant forms are WebAuthn credentials: hardware security keys, and their mainstream successor, passkeys, which live in your phone's or laptop's secure hardware and sign a challenge that's bound to the real site's origin, so a look-alike site gets a signature it can't use. Passkeys have been the industry's default recommendation since around 2023 (Google made them its default sign-in option that October), and the trade-off you'll argue about is enrollment friction. Passwords are "something you know" that everyone ends up knowing. The second factor is what makes the stuffing run worthless, and the origin-bound second factor is what makes the phishing kit worthless too.

One last door in this section that attackers love because engineers forget it: account recovery. Every control above can be bypassed by a support agent who resets a password for someone with a convincing story. Recovery flows need the same rigor as login (verified factors, delays and notifications on high-risk changes, no "security questions" that are public facts), and support staff need a script that doesn't bend. The weakest factor in a system is the human who can override the others.

Section 7 — Bots at the door, floods at the wall

Your traffic spikes fifty-fold in the middle of the night. Is it a viral launch or an attack? The answer decides everything, and the two candidates need different weapons. Bots are application-level abuse (scrapers, stuffers, scalpers buying your inventory), fought with identification and friction. DDoS, distributed denial of service, is capacity-level abuse, traffic whose only purpose is to make the service fall over, fought with absorption and shedding. Confuse them and you'll CAPTCHA a flood (useless) or null-route a product launch (career-limiting; a null route is the network-level "drop everything to this address" that stops an attack by stopping your product).

Bot mitigation: the challenge ladder. The goal is to price bots, not to block them. Each rung costs the bot operator more while costing legitimate users almost nothing:

1. Invisible signals. TLS fingerprinting (real browsers negotiate TLS in recognizable ways; scripts often don't), HTTP/2 behavior, mouse-movement entropy, request timing. The request proves it's a real browser without the user doing anything, and most legitimate traffic clears this rung untouched.
2. Proof of work. A JavaScript challenge that takes a real browser fifty milliseconds and a curl script forever, because it can't run JavaScript. Be clear about what this stops: it filters the cheap, script-only bots. A headless Chrome driven by an automation library like Puppeteer runs JavaScript fine, which is why the ladder has more rungs and why a small computational puzzle per request (trivial for one user, expensive at a million requests) is the other version of this step.
3. Interactive challenge. CAPTCHA, the much-hated, much-needed friction. Modern versions are risk-scored (only the suspicious see them) rather than universal, and that's not just kindness: CAPTCHA-solving farms charge per thousand solves, so showing the challenge only to suspicious traffic raises the attacker's cost per account without taxing everyone.
4. Block. For the traffic that fails everything: no legitimate user behaves like this.

This is where the rate limiting post connects and where it ends: rate limiting is fairness between identifiable clients; bot mitigation is for clients whose identity is the lie. A scraper rotating through ten thousand IPs laughs at per-IP limits, because each IP stays under the limit. The challenge ladder doesn't care about the IP; it cares whether there's a browser behind it. Use both: rate limits for the economics of legitimate use, bot defenses for the adversaries gaming identity itself. (The rate limiting post's Section 8 made the companion point: rate limiting is not DDoS protection either. Three tools, three jobs, though in practice the edge's rate-limiting rules do double as the first response to an application-layer flood.)

DDoS, in two tiers. The tiers come from the network layers: L3/L4 attacks work on packets and connections, L7 attacks work on HTTP requests.

L3/L4 volumetric attacks don't care about your application; they want the pipe full. SYN floods open millions of TCP connections and never finish the handshake. Amplification attacks send small forged requests to public UDP services (DNS, NTP) that reply with large responses to your address. The scale has grown past anything a single organization can absorb: Cloudflare mitigated a 31 terabit-per-second attack in late 2025 and now counts hyper-volumetric attacks (over a terabit per second, or over a billion packets per second) in the thousands per year. You don't fight these on your servers. You fight them with anycast and scrubbing at the edge: the attack traffic gets announced to the nearest of hundreds of points of presence (PoPs) and absorbed and filtered there, the way the CDN post's edge absorbs legitimate flash crowds. Your origin never sees it, provided the origin's own IP address isn't discoverable; an attacker who finds the real address goes around the edge entirely, so the origin should accept traffic only from the edge provider's ranges.

L7 application attacks are smarter and smaller: "slowloris" connections that send headers one byte at a time and never finish, expensive search endpoints hit in loops, login pages hammered. These do reach your application, and the defenses are application-shaped: aggressive caching of the targeted endpoints, shedding at the edge (the edge can say no before your origin spends a millisecond), identifying the expensive paths and making them cheap or authenticated, and the challenge ladder above, since an L7 flood is usually bots.

You will not build DDoS mitigation yourself. The edge providers (Cloudflare, AWS Shield, Akamai) exist because absorbing tens of terabits takes tens of terabits of spare capacity, and you don't have that. AWS Shield's basic tier is free and automatic; the advanced tier is a $3,000-a-month line item. What you do build is the architecture that lets the edge help you: stateless servers behind the edge, cacheable responses, a hidden origin, no origin-only chokepoints, and a runbook (Section 9) for the day the graph goes vertical.

Section 8 — See it coming

Prevention gets the conference talks. Detection is what actually saves you, because prevention fails quietly, partially, and first. The attacker who gets in (and assume they will; the title isn't subtle) should find a system that's watching. What watching looks like, concretely:

Baseline, then notice. You can't spot anomalous without knowing normal. The signals worth baselining: login success rate (Section 6's crater), 404 rate per client (Section 2's enumeration), signup velocity (a thousand new accounts in ten minutes is either a launch or a bot farm, and you know which one you scheduled), traffic mix by geography and by network (by ASN, the autonomous system number that identifies which ISP or hosting provider an address belongs to; a spike from one hosting provider's ASN is a tell), error-rate spikes on expensive endpoints, and egress volume, because data leaving is the thing exfiltration is. An anomaly isn't "a big number." It's "a number that doesn't look like yesterday." Store the baselines, compare continuously, alert on the deviation.

Audit logs are a different thing from application logs, and you need both. An application log says what the code did. An audit log says who did what to which object, when: user 4821 read invoice 9042; admin j.doe changed user 77's email; API key sk_…3f exported 40,000 records. Audit logs are append-only, kept separately from the noisy application stream so they survive log rotation, and they include administrative actions, which are the ones attackers most want to hide. They're the only way to answer Section 9's third step, "what did they touch?", and they're also what compliance regimes mean when they say "audit trail."

Honeypots and tarpits. A honeypot is a door that shouldn't exist: an /admin endpoint with no link to it, a fake credentials file, a .git directory your deploy never creates. No legitimate user ever knocks, so anyone who does is, with very few exceptions (crawlers, your own contracted scanner, a curious employee), a scanner, and you've identified them with almost no false positives. A tarpit is the crueler cousin: instead of rejecting the scanner, it slows them, drip-feeding responses and holding connections open. Every second the scanner spends stuck in your tarpit is a second they're not scanning someone else. Honeypots don't prevent attacks; they convert "are we being probed?" from a question into a feed.

Alert on abuse like you'd alert on an outage. The observability post's (#9) rules apply verbatim: alert on symptoms (login success rate cratered, signup velocity a hundred times baseline), not causes; make every alert actionable with a runbook ("if this fires, block the ASN at the edge, then check the WAF logs"); keep the pager boring. Abuse alerts deserve the same discipline as latency alerts. A stuffing run you sleep through is an outage you chose.

The failure mode here isn't a missing tool. Picture a team with all the signals: the 404 rate per client sits right there in their logs, but there's no baseline and no alert, so nobody is looking. They discover a six-week enumeration of their user IDs from a customer's email: "is there a reason someone's been trying every profile URL?" The data taken was public-ish; the embarrassment was total. Their postmortem doesn't buy anything. It writes five alerts. Detection is mostly the decision to look.

Section 9 — The principal's playbook

Knowing the controls is table stakes. This section is about the calls that separate a system with security features from a system that's actually hard to hurt.

Defense in depth: no single layer holds. Every control in this post fails some of the time. Unguessable IDs leak in a log. A signature scheme has a bug. A WAF rule misses a variant. A human approves the phishing email. So you stack them, and each layer's job is to make the attacker's life harder given that the previous layer failed. The signed URL is checked and the resource requires the right session and the egress is monitored and the anomaly alert fires. The model isn't a wall; it's a maze. The attacker who beats one layer finds another, and every layer they beat costs them time, noise, and evidence. Design each layer as if it's the only one, and assume none of them is.

Abuse economics. You cannot make attacks impossible. You can make them unprofitable, and unprofitable is enough, because attackers are running a business. Every defense in this post is secretly a price hike. Unguessable IDs turn "enumerate everything" from one loop into something like 2¹²² divided by the number of live IDs guesses per hit, which is still astronomical. Proof-of-work challenges turn a million-request flood into a million CPU-seconds the attacker pays for. Risk-scored CAPTCHAs raise the cost per fake account to whatever the solving farms charge. Miss-rate limiting turns an overnight scrape into a months-long one, and a months-long scrape against a target that might fix the hole tomorrow is a bad investment. The defender's goal was never perfect security. It's making the attack cost more than the loot. When you're deciding between two controls, ask which one raises the attacker's cost per attempt more. That's the one that works.

Security is a process, not a launch task. The controls above get built once; the threats keep changing. Three habits keep the posture from rotting. Threat-model per feature, in the design review, with Section 1's three questions, so "how would someone abuse this?" is asked before the code exists rather than after the bill arrives. Run static and dynamic scanners in CI (SAST reads the code for known-bad patterns; DAST attacks a running instance) as a floor, not a ceiling. And publish a way for outsiders to tell you things: a security.txt file, a disclosure policy that promises not to sue people who report responsibly, a triage SLA, and, once you have the muscle, a bug bounty. Section 1's "bored and curious" attacker becomes a free red team the moment you give them somewhere to send the report.

Fail closed, and keep a fire escape. Every control in this post depends on something that can go down: the identity provider, the fraud scorer, the rate limiter's Redis, the WAF's rule feed. Decide in advance what each one does when its dependency is unreachable, because the default is usually the dangerous direction. The rule of thumb: fail closed (refuse) for anything that grants access or moves money, since "we let everyone in while the auth service was down" is a breach with an outage attached, and fail open (allow, and log loudly) for controls whose absence costs you only some abuse for a while, like rate limiters and spam scoring. The resilience post (#2) makes the same decision for every fallback and calls it the fail-open-or-closed decision; here it has a second edge, because attackers watch status pages, and an outage is the moment they test whether your checks are still running. The fire escape is the other half. If your administrators sign in through the same single sign-on as everyone else, a broken identity provider locks the responders out too, which is exactly what happened to Cloudflare's own team in July 2019. So keep a break-glass path: one or two emergency admin accounts with hardware keys, credentials sealed somewhere the on-call can reach without the systems that are down, on an admin plane that lives on its own hostname and network with its own IP allowlist, and an alarm that fires whenever the path is used, because break-glass use is either an incident or an attacker.

The first hour of a real incident. It will happen: the alert fires, the graph is wrong, someone's in. The hour has an order, and the order matters more than the tools:

1. Contain (minutes 0–15). Stop the bleeding with the bluntest instrument available: block the IP or ASN at the edge, rotate the exposed secret, flip the feature flag (the kill switch) that disables the abused endpoint, revoke the leaked tokens. Precision is for later; stop the damage now. This is why the edge controls and kill switches from earlier sections exist. The incident is when you spend them.
2. Preserve evidence (minutes 0–30, in parallel). Snapshot the logs before rotation eats them, save the malicious payloads, note the timeline. You will want all of this for the postmortem and possibly for law enforcement. "We deleted the logs to save disk" is how incidents become mysteries.
3. Assess scope (minutes 15–45). What did they touch? Which accounts, which data, which systems? The audit logs, the honeypot feed, and the anomaly baselines from Section 8 are how you answer. This is the hour your detection investment pays out.
4. Communicate (minutes 30–60). Internal first: leadership and legal, because breach-notification clocks start ticking fast in many jurisdictions (72 hours to the regulator under GDPR, six hours to CERT-In in India, four business days to the SEC for US public companies, "without unreasonable delay" under most US state laws). Then affected users, with specifics rather than reassurance. "We detected unauthorized access to email addresses between Tuesday and Thursday; passwords were not affected" beats "we take security seriously" every time.
5. Eradicate and recover (after the hour). Remove the attacker's foothold, patch the hole, rotate everything they might have touched, then bring systems back deliberately, watching the signals for re-entry.

Run this as a tabletop exercise once a quarter, before you need it. The first time your team argues about who calls legal should not be during a breach.

In an incident, speed of containment beats elegance of diagnosis. You can be wrong about the how for an hour. You cannot afford to be slow about the stop.

Security versus usability: the trade nobody lets you dodge. Every control in this post taxes legitimate users. CAPTCHAs abandon signups, MFA loses the users who won't enroll, aggressive fraud scoring declines good transactions, short session timeouts annoy everyone. A principal doesn't ask "is this secure?" They ask what it costs and what it buys, with numbers on both sides: the fraud rate without the control versus the conversion drop with it. Sometimes the math says the control isn't worth it. A CAPTCHA that stops $200 a month of abuse while costing $20,000 a month in abandoned signups is a bad trade, and saying so is the job. Sometimes the math is existential; the phishing tolerance of a bank is not the phishing tolerance of a meme generator. The right security posture for a system is a business decision, and making it explicitly, with numbers, in a review, before the incident, is what separates judgment from security theater.

The closing reframe: junior engineers add security features. Senior engineers build detection. Principals do the economics: price the attack above the loot, stack the layers so each failure is survivable, and decide the trade-offs with numbers before the version of them that happens during an incident has to be decided with adrenaline. Assume they're already knocking. Build like you believe it.

Assume They're Already Knocking, distilled

For the skimmers and the revisitors: everything above, on one page.

Key numbers and rules:

The decisions, in one table:

Three ideas to take with you:

1. "Unlisted" is not "private," and hiding an ID is not the same as checking who's allowed to see it. Anything an attacker can guess, they will, in a loop, a hundred thousand times. Unguessable IDs for links that are their own permission, an authorization check for everything else, and 404s that leak nothing.

2. Every defense is a price hike. You will never make attacks impossible; attackers are running a business and you are in their cost column. Unguessable IDs, proof-of-work, risk-based challenges, miss-rate limits: each one raises the cost per attempt. Price the attack above the loot and the rational attacker goes elsewhere.

3. Prevention fails; detection saves you. Every layer breaks some of the time. Assume it, stack the layers, and invest in seeing: baselines that notice, audit logs that answer "what did they touch," honeypots that confirm, alerts with runbooks. The team that detects in minutes contains in an hour. The team that doesn't reads about it in a customer email.

Further reading

• OWASP Top 10 (2025). The industry's shared list of the most critical web application risks; broken access control at #1 and the broadened supply-chain category are Sections 2 and 4.
• OWASP API Security Top 10. Where IDOR/BOLA lives at #1; the authorization half of Section 2.
• OWASP Cheat Sheet Series. Concrete defensive patterns: input validation, secrets management, SSRF prevention, authentication, password storage; the practitioner companion to Sections 3–6.
• RFC 2104: HMAC. The message-authentication construction behind Section 3's signed URLs; short, precise, worth one read.
• Stripe: Webhook signatures. The production-grade version of Section 3's recipe, including secret rotation.
• Have I Been Pwned: Pwned Passwords API. Breached-password screening with k-anonymity; behind Section 6.
• NIST SP 800-63B: Digital Identity Guidelines: Authentication. The source for "check breach lists," "SMS is restricted," and "OTP is not phishing-resistant."
• Cloudflare Learning Center: What is a DDoS attack. The tiers and the absorption model; behind Section 7.
• Microsoft PhotoDNA. Perceptual hashing for known-bad image detection; the technology behind Section 5's hash-check stage.
• Sqids. The ID-obfuscation library whose own documentation explains, correctly, why it isn't encryption; behind Section 2.

Where you'll meet this

This post is the interview's adversarial turn, finally answered: the sequential-keys question from the URL shortener's principal round now has nine sections behind it, and Step 11's keyed-permutation answer is Section 2 here at full length. It leans on the rate limiting post (#8) three times: miss-rate limiting against enumeration in Section 2, the choice of limiting key in Section 6, and the three-tools-three-jobs split in Section 7. The edge-absorption model in Section 7 is the CDN post's (#7) geometry pointed at hostile traffic, and the "trust only your own hop's X-Forwarded-For" rule is the load balancer post's (#11) Section 3 seen from the attacker's side. Abuse alerting in Section 8 runs on the observability post's (#9) rules. The signed-webhook pattern in Section 3 is where this post meets idempotency (#6), because a webhook that's verified still has to be safe to receive twice. Next up is estimation (#13), because every defense in this post has a cost, and the numbers are how you decide which ones to buy.

Keep exploring: the Core Concepts series

Every post in the series stands alone. Read them in any order.

• From Laptop to Billions of Clicks: Designing a URL Shortener in 11 Steps — the anchor: one design, every concept under load.
• #1 The 100:1 Superpower: Caching — the fastest request is the one you never make.
• #2 The Blast Radius: Surviving the Day Your Dependencies Fail — staying up when everything you depend on goes down.
• #3 Divide and Conquer: Sharding and Partitioning — splitting one database into many without losing your mind.
• #4 The Snowflake Problem: Unique IDs at Scale — naming things when millions are born every second.
• #5 Copies of the Truth: Replication — keeping copies of your data that actually agree.
• #6 Do No Harm Twice: Idempotency — making "just retry it" safe.
• #7 The Copy at the Doorstep: CDNs and Edge Computing — serving from next door instead of across the ocean.
• #8 The Bouncer's Math: Rate Limiting — saying no politely, at scale.
• #9 What Broke at 3 AM: Observability, p99, and Useful Alerts — knowing what's wrong before your users tell you.
• #10 Do It Later, On Purpose: Async Processing and Queues — the work the user doesn't have to wait for.
• #11 The Traffic Cop: Load Balancing — the box in every diagram nobody explains.
• #12 Assume They're Already Knocking: Security and Abuse at Scale — designing for the users who are designing against you. (this post)
• #13 Do the Math First: Estimation for System Design — the two minutes of arithmetic that choose the architecture.
• #14 The Whiteboard Playbook: Taking On Any System Design Challenge — the whole series in forty-five minutes.

Blueprints of Scale — Core Concepts #12. If this helped, the best thanks is a share with someone who's learning.

The short version: a CDN (content delivery network) is a fleet of servers scattered across the planet, each holding copies of your content, so users download from the one in their neighborhood instead of the one on your continent. Edge computing is the sequel. Once you have computers in every neighborhood, you start running code on them (auth checks, A/B tests, personalization) instead of just serving files.

Here's what's covered: the night a viral photo melted an origin server; what a CDN actually is (points of presence, edge caches, why distance is the real latency, and how a CDN differs from a reverse proxy); hit ratio, the metrics around it, and how to measure what users actually experience; TTLs, the Cache-Control directives everyone misreads, conditional requests, and why purging is the hard part; what the edge can and can't cache, including private content, negative caching, video, compression, and HTTP/3; how requests find the nearest edge (anycast and DNS), TLS at the edge, geography and compliance; tiered caching and the origin shield; the failure modes (stale content, poisoning, stampedes, and the origin going away); when not to use a CDN; and the principal-level toolkit: edge compute and its limits, consistent hashing, stale-while-revalidate, the real cost model, and multi-CDN.

If PoP and anycast are new words, start at Section 1; the first two sections assume nothing, and terms are defined as they appear. Sections 3 through 8 are the machinery and the failure modes. Section 9 goes deep. The cheat sheet is at the end under The Copy at the Doorstep, distilled, and every diagram is also described in the text around it.

Section 1 — The night the origin melted

A small photo-sharing app. One server in Virginia on a 1 Gbps link, humming along at a few hundred requests a second. Then a user posts a photo of a dog in a raincoat, a celebrity reposts it, and within the hour it's the most-viewed image on the internet that day.

Two million requests for the same 400-kilobyte JPEG in the first hour, from six continents. That's 800 GB in an hour, about 1.8 Gbps sustained, on a link that carries 1. The server did the only thing it knew: read the file from disk and send it, two million times, across oceans. The link saturated, requests queued, the site went down. Nothing was broken. The architecture just assumed every user would come to Virginia.

The part that stings: the file never changed. Nothing about the request was personal. Every user in Tokyo waited something like 170 milliseconds for the round trip across the Pacific and back (light in glass accounts for about 110 of them; routers and queues for the rest), plus whatever the origin's queue added, when a copy sitting in Tokyo would have answered in about 20. This was a geography failure, not a capacity failure. You cannot buy a server big enough to beat the speed of light.

Two universes. Without the CDN, all two million requests travel to Virginia and back, and the origin does the same work until it falls over. With it, the first Tokyo request fetches the photo once; the Tokyo edge keeps a copy; the next million Tokyo requests never leave Tokyo.

Once per city instead of once per user. That's the entire economic argument in one ratio. A CDN doesn't make your server faster; it makes your server unnecessary for exactly the requests it was worst at: identical, cacheable bytes, wanted by faraway strangers, all at once. The origin becomes the source of truth almost nobody talks to directly, the filing room the whole world stopped visiting because every neighborhood got its own desk.

Section 2 — What a CDN actually is

A CDN has three pieces at its core, and everything else in this post is built on top of them.

1. The origin. Your server, unchanged. It holds the original files. Once a CDN sits in front of it, the origin's job shrinks to "answer the questions nobody else can," and its traffic drops by 90% or more.

2. Points of presence (PoPs). Data centers the CDN company operates in hundreds of cities. "Point of presence" is industry-speak for "a building full of servers in a city." Cloudflare is in around 350 cities as I write this (call it three hundred-odd for the arithmetic below); the other big networks look similar. Each PoP holds edge caches: servers whose whole job is storing copies of content and serving them fast.

3. The routing layer. The machinery that sends each user to their nearest PoP; Section 6 takes it apart properly. For now: you type a URL, something figures out which PoP is closest, and sends you there.

The request path, step by step. A user in Berlin opens your page, which references hero.jpg:

Read it top to bottom. The user is routed to Frankfurt; the edge cache checks its shelves. On a hit, the bytes never leave Germany: about 15 milliseconds. On a miss, the request climbs one rung to a larger upper-tier cache (Section 7 explains those), and then to the origin as the last resort. And the move that makes it all work: every miss fills the shelf on the way back. The origin answers once, the upper tier keeps a copy, the edge keeps a copy, and the next thousand Berliners never climb past Frankfurt.

A useful way to place the CDN among the things it resembles: a single nginx or Varnish server in front of your app is a reverse proxy cache, one node that does exactly what the Frankfurt edge does. A CDN is a reverse proxy cache distributed across hundreds of cities with routing that sends each user to the nearest one. And what the vendors now call an edge platform is a CDN plus compute (Section 9) plus a security layer (Section 6). Same idea at three sizes.

So when do you actually need one? The day your users sit farther from your origin than your patience for latency; for a public website, that's day one. One office on the same network as your servers? Skip it.

Two details worth pocketing before we move on.

Distance is latency, and latency is physics. Light in fiber travels about 200 kilometers per millisecond (it would be faster in a vacuum, but the internet runs on glass). Virginia to Tokyo is about 11,000 km along the great circle, so 55 ms one way at the speed of light in fiber, before routers, queues, and your application add their share. Measured round trips between a Virginia cloud region and Tokyo run around 150 ms, and a real user's last mile adds more. A PoP 50 km away answers in single digits. No amount of code optimization beats moving the bytes closer. The CDN is not a performance tweak but a different geometry.

The CDN sees your traffic before you do. Every request passes through the edge first. That position, in front of everything, is what makes Sections 6 and 9 possible: TLS termination, DDoS absorption, and eventually running your code there. The edge isn't just a cache with a good address; it's the front door of your system.

Section 3 — Hit ratio: the one number

If you remember one number from this post, make it this one. The caching post had its hit ratio; the CDN's is the same number at a larger scale. Hit ratio is the percentage of requests the edge answers from its own shelves without climbing to the origin. 1,000 requests arrive in Frankfurt, 970 never leave Germany, 30 get fetched from Virginia: 97% hit ratio. The origin handled 30 requests instead of 1,000.

Why this number runs everything: every point of hit ratio is origin capacity you don't have to buy. At 90%, your origin handles a tenth of the traffic. At 99%, a hundredth. The gap between 95% and 99% doesn't sound like much until you translate it: it's a 5× reduction in origin load. And for the CDNs that bill per gigabyte (CloudFront, Fastly, Akamai), edge-served gigabytes end up cheaper than origin-served ones once your volume reaches the CDN's steeper tiers, because the CDN's business is buying internet transit in bulk (Section 9 has the real prices, and at small volumes the saving is mostly the free origin-to-CDN transfer). Hit ratio is a latency metric, a capacity metric, and a money metric at the same time. That's why it's the one number.

It comes in three flavors, and the dashboards mix them up. Request hit ratio is the count above: what fraction of requests were hits. Byte hit ratio is what fraction of bytes were served from cache, and it diverges from the request ratio when large objects miss; a 99% request hit ratio with every 2 GB video missing is a bad day for the origin. Origin offload is what the vendors usually mean by "offload," and it's typically the byte version. Watch the request ratio for latency and the byte ratio for bandwidth and cost.

Rules of thumb, not benchmarks, because the right number depends entirely on what you serve:

Three levers move the number, in order of leverage:

1. What's cacheable at all. A site serving the same hero image to everyone can hit 99%. A site where every page is personalized per user might struggle to break 50%. The content mix sets the ceiling, and Section 5 is the whole topic.

2. TTLs. Longer time-to-live means entries survive longer, so more requests hit. A one-hour TTL on a logo that changes yearly is leaving money on the table; a one-year TTL is nearly free offload. The price of long TTLs is staleness, which is Section 4.

3. Cache key design. The edge decides "same or different" by the cache key: usually the URL, but it can include headers, cookies, or query strings. Include too much (a per-user cookie) and every user gets their own copy, so nothing is ever "the same" twice and the hit ratio collapses. Include too little and users get each other's content, a correctness bug and the subject of Section 8's poisoning story. The cache key is where performance and correctness shake hands, and getting it right is most of the fiddly work of running a CDN:

• Normalize the URL before keying: lowercase the host, strip the trailing slash, sort query parameters, so /Page?b=2&a=1 and /page/?a=1&b=2 are one entry, not two.

• Allowlist query parameters that change the response and strip the rest. Marketing tags like ?utm_source= are the usual leak: same page, dozens of entries. The default differs by vendor and both directions bite. Cloudflare keys on the full query string by default, so tracking parameters fragment the cache; CloudFront's "caching optimized" policy ignores query strings entirely, so ?page=2 gets served page 1 unless you tell it otherwise.

• Strip cookies and headers from the key unless the response depends on them, and if it does, key on the specific value (the language, the device class, the country) rather than the whole cookie jar.

Beyond the ratio, three more things belong on the day-one dashboard: the origin error rate (a spike means misses are turning into failures), the cache status of individual responses (every CDN adds a header, CF-Cache-Status, X-Cache, and the states HIT, MISS, EXPIRED, REVALIDATED, BYPASS, DYNAMIC are how you answer "why is this URL a miss" in one request), and what users actually experience. That last one needs real user monitoring (RUM): a small script in the page reporting timings like time-to-first-byte by geography, because the PoP that served a user isn't always the nearest one, and the only way to know is to measure from the user's side. Synthetic probes from a few regions are the cheap complement.

Alert on sudden drops in hit ratio. A ratio falling off a cliff means the edge stopped absorbing load (a purge gone wrong, a deploy that renamed URLs, a TTL someone fat-fingered) and the origin is about to feel the full weight. Read the ratio before you scale the origin.

Section 4 — TTLs and the purge problem

There are only two hard things in computer science: cache invalidation and naming things. The joke survives because the first half is barely a joke.

TTL first, because it's the easy half. Time-to-live: how long the edge keeps a copy before asking the origin for a fresh one. hero.jpg, TTL one day: the Frankfurt PoP fetches it Monday morning and serves it from the shelf until Tuesday morning. Long TTL means a sky-high hit ratio with slow-propagating changes; short TTL means fresh content with a harder-working origin. Same dial as the caching post's (#1) Section 4. A TTL is a promise about how wrong you're willing to be, and for how long.

Election night, CDN edition. A news site caches its homepage at the edge with a 30-minute TTL, which is perfectly sensible on a calm Tuesday. The race is called at 11:04 PM. Until as late as 11:34, depending on when each PoP last refreshed, every PoP on earth serves "Too close to call." The CDN worked exactly as configured, and that was the problem: the TTL was a performance setting chosen on a quiet day, and it became an editorial decision on the loudest night of the year.

The headers that set the TTL, since they're misread constantly. The origin sets the TTL through the Cache-Control response header, and the directives look alike while meaning very different things:

The pair that causes the most bugs is no-cache versus no-store. The pair that causes the most missed opportunity is max-age versus s-maxage, because it lets you cache aggressively at the CDN (where you can purge) while keeping browser copies short (where you can't); the one thing to know before leaning on it is Section 9's note that s-maxage and stale-while-revalidate don't combine.

Conditional requests, the cheap refresh. When a cached copy expires, the edge doesn't have to re-download it. If the origin sent an ETag (an opaque version identifier for the response) or a Last-Modified date, the edge sends the origin a request with If-None-Match: <that etag>, and the origin answers with a bodiless 304 Not Modified if nothing changed. The copy's freshness is renewed for the price of a few headers. Two traps: a "weak" ETag (prefixed W/) says the content is semantically equivalent, not byte-identical, and some caches won't use it for byte-range requests; and if you have several origin servers behind a balancer that each compute the ETag from local file metadata (older Apache defaults folded in the file's inode number, which differs per server), the same file has a different ETag on every server, every revalidation fails, and you re-download forever without noticing. Make ETags a function of the content.

So you need the other half of the joke: purging (also called invalidation), telling the CDN "forget your copies of this URL right now, everywhere." You push the new homepage, purge /index.html, and the next request in every city fetches fresh. One API call. What's hard about that?

Three things.

"Right now" is a range. A purge has to reach hundreds of PoPs. The good news is that the big networks are fast: Fastly and Cloudflare propagate a single-URL purge globally in well under a second, Akamai in seconds, and CloudFront's tag-based invalidations take effect within about five seconds and complete within thirty. The bad news is that it's never atomic. During that window, some cities serve the new page and some serve the old one, and your deploy is simultaneous nowhere. Most days nobody notices. During a breaking-news correction or a security fix, "most days" isn't the standard you want.

Purging is a blunt instrument at scale. Purging one URL is fine. Purging a million (a site redesign, a mass takedown) runs into rate limits before anything else: the limits vary wildly by vendor (Cloudflare's free plan takes 800 URLs a second in batches of 100, CloudFront 150 paths a second with a bill past the first thousand, Fastly around 100,000 single-URL purges an hour), so a million single-URL purges takes anywhere from twenty minutes to ten hours, plus the origin hammered as every PoP refetches at once (the thundering herd, coming up in Section 8), plus some CDNs charge per path beyond a free allowance. The lesson is not "purge faster." It's "purge by tag or prefix," and it's the next paragraph but one.

The grown-up answer is: don't purge, version. Instead of overwriting app.js and purging it, ship app-2f9a1c.js, a new filename with the content hash (a fingerprint computed from the file's bytes) baked in. The HTML points at the new name; the old file sits in caches until its TTL dies of old age, harming no one. Immutable assets with hashed filenames never need purging, because a changed file is a different URL. Every serious frontend build pipeline fingerprints its assets for exactly this reason. The HTML shell that references them is the one thing you can't version (its URL is /), so it gets the opposite treatment: a short max-age or no-cache with an ETag so it revalidates cheaply, and never a long TTL; index.html is the one file you never long-cache. (If you ship a service worker, it's one more cache in the chain, with its own rules; it can hold a stale shell for as long as its script says to.)

That's the decision you'll make on every deploy: versioned filename (purge-free, the happy path) versus purge (necessary, slightly ragged).

The working setup: versioned filenames for everything with a build step (JS, CSS, bundled images). Short TTLs (minutes, not hours) plus stale-while-revalidate for the unversionable shell and for API responses. Purge as the fire extinguisher: present, tested, rarely used. And test the extinguisher before the fire: run a purge in staging and time what your CDN actually takes, because the number differs by vendor and plan.

Purge by tag, not by URL. When a product's price changes, you don't want to purge 40,000 URLs that mention it. Tag-based purging flips it around: the origin stamps responses with tags (product:4821 on the product page, the category page, the search results, the homepage module), and one purge call for the tag invalidates all of them at once. Fastly calls them surrogate keys, Cloudflare and Akamai call them cache tags, and CloudFront added tag invalidation in 2026. One API call, thousands of entries, consistent everywhere. This is how real sites purge at scale: not URL by URL, but family by family. And when the change is "everything under /docs/," a prefix or wildcard purge does the same job.

One more purge idea, from Fastly's vocabulary: a soft purge marks entries stale instead of deleting them, so the next request serves the old copy while fetching the new one, exactly like stale-while-revalidate. A hard purge on a hot key at peak creates the stampede Section 8 describes; a soft purge doesn't.

What about pre-loading the cache? You can't warm 300 PoPs; the first user in each city always misses. What you can do is warm the upper tier (Section 7), which is one place, by crawling the URLs you know will be hot before a launch. After that, tiering is what turns one cold miss into one origin fetch instead of hundreds.

The mark of a mature CDN setup is needing almost no purges, not having fast ones.

Section 5 — What the edge can and can't cache

Not everything belongs on the edge's shelves. Sort your traffic into three buckets; every CDN deployment starts here.

Bucket 1: static and identical for everyone. Images, videos, fonts, JS, CSS, PDFs. Same bytes for every user, changing rarely. This is the CDN's home turf: long TTLs, versioned filenames, hit ratios in the high nineties. If your site is slow and this bucket isn't on a CDN, stop reading and go fix that first.

Bucket 2: shared by many, changing slowly. An API response like "today's headlines," a product listing page, a logged-out homepage. Same for large groups, but not forever. Cacheable with short TTLs (a minute, five minutes) and, as the default posture, stale-while-revalidate and stale-if-error on top, so expiry never makes a user wait and an origin outage never makes them see an error. The edge absorbs the flood; staleness is bounded and boring. Most of the interesting CDN work lives here: the art is finding the parts of "dynamic" pages that are actually the same for everyone, and caching those parts aggressively.

Bucket 3: truly personal or truly real-time. Your bank balance, a live auction bid, a shopping cart. Different for every user, wrong if stale. The edge must not cache these; serving someone else's bank balance is the nightmare from Section 3's cache-key warning. These requests pass through the CDN to the origin, marked private or no-store.

Bucket 3 has a sub-bucket that trips people up: content that's private but static, like a paid video or a user's own uploaded photos. The bytes are cacheable; the permission isn't. The pattern is to let the edge cache the bytes and check the permission itself, with a signed URL or signed cookie (CloudFront's signed URLs, Akamai's and Fastly's token authentication, or a small edge function from Section 9 verifying an HMAC (a keyed hash only your servers can compute); the security post, #12, has the recipe). One caveat that cuts both ways: the HTTP caching standard says a shared cache must not store a response to a request that carried an Authorization header unless the response explicitly allows it, so authenticated-but-shared content needs explicit configuration on standards-following CDNs, while CloudFront has the opposite trap, stripping the header from GET requests before forwarding them and caching the response under the shared key unless you tell it to forward the header. And the reverse trap, cache deception: an attacker tricks a victim into loading /account/settings/style.css, the origin ignores the fake suffix and returns the victim's private page, and the CDN, seeing .css, caches it under a shared key for the attacker to fetch. The fix is the same as Section 8's poisoning fix: cache only what the origin's headers say is cacheable, never what the URL's extension suggests.

What people miss: "not cached" doesn't mean "not helped." Even for Bucket 3, the CDN accelerates the connection itself:

• TLS termination at the edge (more in Section 6): the expensive cryptographic handshake happens in Frankfurt, not Virginia. The long-haul leg reuses a warm, persistent connection.

• Protocol optimization: the edge speaks HTTP/3 to the user over the short hop, and multiplexes many users' requests over a few fat, tuned connections back to the origin. TCP's cautious ramp-up happens on the short leg where it costs milliseconds, not the long one where it costs hundreds. HTTP/3 runs on QUIC, a UDP-based transport that folds the TCP and TLS handshakes into one round trip (zero, for returning visitors), doesn't stall every stream when one packet is lost, and survives the client's IP changing mid-connection when a phone moves from Wi-Fi to cellular. Almost no origins speak it; the edge is where users get it.

• Dynamic acceleration: the CDN routes around internet congestion in real time, picking the fastest path to your origin the way a satnav routes around traffic. This is usually a paid add-on (Akamai's SureRoute, Cloudflare's Argo), not a default, and worth the money mostly for origins that are far from most users.

Read it as a sorting exercise for every URL you serve. The beginner mistake is treating the CDN as a binary switch, cached or not. The real deployment is a spectrum: each bucket gets its own TTL, its own cache-key rules, its own purge policy. A CDN configuration is this diagram, written down per URL pattern.

Four more things that live in this section because they decide what the edge does with a request:

Negative caching. CDNs cache errors, too, and by default: CloudFront keeps a 404 or a 5xx for ten seconds unless you say otherwise (it used to be five minutes, and some configurations still are). That's usually right (a missing image shouldn't hammer the origin) and occasionally disastrous: a deploy that briefly 404s an asset while files are still uploading pins that 404 at the edge for five minutes after the file arrives. Set error TTLs deliberately, short for 5xx and moderate for 404, and know that a purge clears them too.

Compression. The edge compresses text responses with Brotli or gzip, and stores the compressed and uncompressed variants as separate cache entries keyed on the Accept-Encoding header (this is the one Vary value every CDN honors). Normalize that header at the edge so gzip, deflate, br and br, gzip don't become two entries, and don't compress twice: an origin that gzips and an edge that gzips again produces bytes nobody can read.

Images and video. Serving the right image format is a negotiation on the Accept header (a browser that accepts AVIF gets a smaller file), and the edge is the right place to do the conversion and resizing (Cloudflare Images, Fastly Image Optimizer, Akamai Image Manager), keyed per variant. Video is served as small segments (HLS or DASH playlists pointing at a few seconds of media each), and segments are perfectly cacheable Bucket 1 objects; large single files are fetched and cached in byte ranges so a user seeking to minute 40 doesn't wait for minutes 0 through 39.

Long-lived connections. WebSockets, server-sent events, and long polling pass through the CDN to the origin; the edge can't cache a stream. What it can do is terminate TLS and hold the connection, and what it will do is impose idle timeouts and sometimes buffer responses, so check both settings before you wonder why your streaming endpoint stalls every hundred seconds.

Start with Bucket 1 (pure win), then hunt Bucket 2 for the biggest traffic you can tolerate being slightly stale, usually API list endpoints and landing pages. Keep Bucket 3 passing through but accelerated. And revisit the sorting every year: features drift between buckets as products change. The page that was Bucket 1 in 2024 is Bucket 2 now, and nobody updated the config. The buckets will lie to you if you let them.

Section 6 — How requests find the nearest edge

The cleverest load balancer on the internet is the internet itself.

Two mechanisms route users to their nearest PoP, and most CDNs use both.

DNS-based routing is the older one. You look up cdn.example.com; the CDN's authoritative DNS server (the one that holds the real answer for that name, as opposed to the resolver your laptop asks) sees where the request came from and answers with the nearest PoP's IP. Berlin asks, gets Frankfurt. Simple and flexible: the CDN can weigh load, drain PoPs, steer traffic however it likes. The catch: DNS answers get cached by resolvers, so steering reacts in minutes, not seconds, and the location signal is the resolver's address, not the user's. A user in Berlin whose company resolver sits in Virginia gets sent to Virginia. The fix for that is EDNS Client Subnet (RFC 7871), where the resolver passes along part of the user's address; Google's and OpenDNS's public resolvers do, and Cloudflare's 1.1.1.1 deliberately doesn't, for privacy.

Anycast is the beautiful one. The same IP address is announced from every PoP simultaneously. Three hundred cities all claim "I am 104.16.0.1." The internet's routing protocol, BGP (Border Gateway Protocol, the way networks tell each other which IP ranges they can reach), doesn't care about intentions; it delivers each packet to the nearest announcer, where "nearest" means the path BGP prefers (each network's own policy first, then the fewest networks traversed), not fewest kilometers. A packet from Berlin flows to Frankfurt; a packet from Tokyo flows to Tokyo. The hostname still resolves to an IP through DNS, but the steering logic and its cache staleness vanish. The internet's own routing table becomes your load balancer.

Two superpowers in one diagram. First, proximity routing for free: both users address the same IP, and BGP's shortest-path logic sorts them to their nearest PoP. (With two caveats: "nearest by path" sometimes lands a user on a PoP that isn't the closest by geography, and a route change mid-connection can break a long-lived TCP session, since the packets start arriving at a different PoP.) Second, the one that pays for the whole CDN: DDoS absorption. Attack traffic, like legitimate traffic, lands on the nearest PoP. A multi-terabit attack doesn't converge on your origin; it dilutes across 300 cities, each absorbing its local share, and each PoP scrubs the junk before forwarding anything. Anycast turns the attacker's geography against them. That covers the volumetric attacks that work at the packet and connection level (the L3/L4 floods, in the security post's terms). Application-level attacks (a loop hammering your search endpoint) reach the edge as ordinary-looking HTTP, and the edge answers them with its other tools: a web application firewall (a WAF, which pattern-matches requests against known attack signatures), rate limiting rules, and bot management. The security post (#12) is where those live. One condition makes all of this work: the origin's real IP address must not be discoverable, or the attacker simply goes around the CDN. Accept traffic at the origin only from the CDN's published address ranges, or use the CDN's authenticated-origin-pull feature.

The second half of the routing story: TLS termination at the edge. Your traffic is HTTPS, so every connection starts with a TLS handshake, cryptographic negotiation that costs one round trip with TLS 1.3 (two with 1.2), on top of TCP's own round trip. Berlin to Virginia is a round trip of 100 ms or so, so that's 200 to 300 ms of handshakes before any data moves. The CDN terminates TLS at the Frankfurt PoP instead: the handshakes happen over the 15 ms hop, and the Frankfurt-to-Virginia leg reuses a long-lived, already-encrypted connection shared across thousands of users. The user pays the handshake cost of talking to their neighbor, not another continent. Certificate management (issuance, renewal) also collapses to one place at the CDN, instead of every origin server you run.

You don't choose anycast versus DNS; your CDN does, usually both. What you choose is whether your DNS lets the CDN do its job: point your domain at the CDN and let it steer. The decision that actually matters is how the CDN talks to your origin. Every vendor has a setting for it, and Cloudflare's names are the ones people know: Flexible (HTTPS to the user, plain HTTP to the origin), Full (encrypted to the origin but the origin's certificate isn't checked), and Full (strict) (encrypted and verified). Always the strict one. The looser modes leave the Frankfurt-to-Virginia leg unencrypted or unverified, which surrenders the privacy your users think the padlock promises. CloudFront verifies the origin's certificate whenever the origin protocol is HTTPS, so there it's a matter of not choosing HTTP.

Since the edge knows where every user is, it also knows where they aren't allowed to be, and where their data is allowed to go. Every CDN adds a country header to requests (CF-IPCountry, CloudFront-Viewer-Country), which is how you localize prices and languages without a database lookup, and every CDN can block or redirect by country, which is how you honor sanctions and licensing. Data residency is the harder version: some regulations and contracts require that certain users' data be processed only in certain regions, and a CDN that caches responses in 300 cities is, by default, storing copies everywhere. The vendors sell region-restricted caching for this. And one geography is its own project: serving users inside mainland China well requires a license (an ICP filing) and a CDN with nodes inside the country, because traffic crossing the border is slow and filtered.

Section 7 — Tiered caching and the origin shield

Three hundred edge PoPs. One purged hot key. One very bad minute.

Section 2's diagram had three rungs: edge, upper tier, origin. The middle rung isn't decoration. Picture the World Cup final's live-score JSON, cached at 300 edge PoPs. Someone purges it; the score changed. Within seconds, all 300 PoPs miss at once, and all 300 ask the origin for the same file. The purge turned your shield into a 300-way stampede: the caching post's thundering herd, at continental scale.

The fix is tiering: edge PoPs don't talk to the origin; they talk to a smaller set of upper-tier caches, and only those talk to the origin. The last tier before the origin is the origin shield: one cache cluster placed near the origin, whose job is to collapse duplicate misses so the origin does the work once. Three hundred edges miss, a dozen upper-tier caches miss, one request reaches the origin. The shield refills, the upper tier refills from the shield, the edges refill from the upper tier.

Follow the misses upward: three edges miss, three regional tiers miss, and the shield lets exactly one request through. On the way back, every layer keeps a copy. The shield is request coalescing (caching post, #1, Section 5: one request goes to fetch, everyone else waits for that answer) at planetary scale.

How the vendors actually shape this, since the diagram simplifies: the regional tier is usually automatic and per-continent (CloudFront has about fifteen regional edge caches between its PoPs and your origin), and the shield is a single location you pick close to your origin (one Origin Shield region per origin on CloudFront, one shield PoP per backend on Fastly, one upper-tier location nearest the origin with Cloudflare's tiered cache). The shield is one place by design: its whole purpose is to be the one funnel.

Coalescing also happens inside each PoP. When fifty requests for the same missing key arrive at Frankfurt in the same second, a well-built edge sends one to the upper tier and parks the other forty-nine until it returns (nginx's proxy_cache_lock and Varnish's waiting list are the same mechanism on one box). The failure mode is worth knowing: when the origin is slow, every parked request is exactly as slow as the one it's waiting on, so coalescing turns "one slow fetch" into "fifty slow responses." Pair it with stale-while-revalidate (Section 9) so the parked requests get the old copy instead of waiting.

Two things to know when you're configuring it.

Every miss now takes one more hop. A shield adds latency to the requests that miss all the way through, and it isn't free on most networks: CloudFront bills per request that reaches the shield, Fastly bills shield-to-edge traffic as ordinary requests and bandwidth, and only Cloudflare's tiered cache is included on every plan. For nearly all origins the trade is obviously worth it, but it's a trade, and you should look at the bill the month after you turn it on.

Tiering also buys you purge sanity. Purge the hot key: edges drop it, the upper tier drops it, the shield refetches once, everyone refills from the shield. Without tiers, the purge is the stampede. With tiers, it's a ripple that dies at the shield.

If your CDN offers an origin shield, turn it on. It's one setting, and your origin stops fearing purges.

Section 8 — Failure modes: stale, poisoned, herded, gone

Four ways the edge breaks. Each one happened to someone.

Stale content that won't die. You purged the bad price ($49 showing as $94) and the purge API said success. But a PoP in Sydney keeps serving $94. Purge propagation isn't atomic (Section 4), and worse, there's a downstream cache you forgot about: a corporate proxy, an ISP cache, a user's browser holding a long-TTL copy, a service worker. None of them got the memo. The edge is not the only cache in the path. Debugging stale content means walking the full chain (browser, ISP, CDN edge, upper tier, shield, origin) and asking each layer what it holds and when it expires; the cache-status headers from Section 3 are how you interrogate the CDN's layers. The fix is procedural: short TTLs on anything that changes, so staleness self-heals in minutes; a longer edge TTL than browser TTL (s-maxage, or the vendor's edge setting when you also want stale-while-revalidate) so the browser's copy is shorter than the CDN's; versioned filenames for assets, so "stale" is impossible; and a purge you've tested in staging before you need it in an incident.

Cache poisoning: the edge serving the attacker's content to everyone. The scary one, and the research that made it famous is James Kettle's 2018 "Practical Web Cache Poisoning." The edge decides "same or different" by the cache key (Section 3). Suppose your key is just the URL, but your origin varies its response on a header, say X-Forwarded-Host. An attacker requests / with X-Forwarded-Host: evil.com. The origin, trusting the header, renders a page full of links to evil.com. The edge caches it under the key /. Now every visitor gets the attacker's page. The cache did its job perfectly. The key was wrong; it didn't include the thing the response varied on.

One crafted request, one poisoned entry, every subsequent visitor served the attacker's content until the TTL expires. The defense is a rule you can audit: the cache key must include everything the response varies on. If the origin varies on a header, that header goes in the key, or better, the origin stops varying on untrusted input. The paranoid version: don't cache responses that vary on any request header you didn't explicitly allowlist.

A word on the Vary header, because it's the mechanism people expect to handle this and mostly it doesn't. In the HTTP standard, an origin sends Vary: Accept-Language to say "this response depends on that header; cache one copy per value." Browsers honor it. Most CDNs do not, except for Accept-Encoding: Cloudflare ignores other Vary values by default, CloudFront honors only the headers you've added to the cache policy, and Akamai's default is to treat anything else as uncacheable. Vary: Cookie makes a response effectively uncacheable everywhere. The CDN's way of saying "one copy per language" is a cache-key rule, which is Section 3's list. Set the key; don't rely on Vary.

The purge stampede. Section 7 set this one up: purge a hot key at peak, every PoP refetches at once, the origin melts. The shield tames it; a soft purge or stale-while-revalidate avoids it. Without either, stagger your purges, and never hard-purge your hottest keys during peak unless the content is actively harmful. Purging is a write operation against your own infrastructure. Treat it with the respect you'd give a deploy.

The origin goes away. Every design above assumes the origin eventually answers. When it doesn't (a bad deploy, a database failover, a region outage), the CDN can be the difference between "the site is down" and "the site is a little stale." Two settings do it. stale-if-error (Section 4) tells the edge to keep serving the copy it has when the origin returns an error, for as long as you allow. And origin failover (CloudFront origin groups, Fastly's multiple backends, Cloudflare's load balancing) gives the CDN a second origin to try when the first fails its health check, which is the load balancer post's (#11) job done at the CDN. Set both, and test the failover on a Tuesday.

Since the edge now handles most of your traffic, your origin's logs no longer describe your traffic; they describe the misses. Analytics and debugging move to the edge: every CDN offers real-time log delivery (CloudFront real-time logs, Cloudflare Logpush, Fastly's log streaming), usually sampled, usually billed, and the observability post (#9) is where those logs go.

And the question, asked properly: when should you not use a CDN?

• Your users are next to your servers. Internal tools, one-office companies, a game where players connect to regional servers directly. If the farthest user is 20 ms away, the CDN's 15 ms buys nothing.

• Everything is Bucket 3. A real-time trading terminal, a multiplayer game server, an API where every response is per-user and per-second. The CDN can still accelerate the connection (Section 5), but if nothing is cacheable, you're paying for a very expensive TCP optimizer. Do the math first.

• You're pre-product-market-fit and broke. A CDN costs real money at scale, and configuration time always. A side project with 100 users is fine on one server. Add the CDN when the traffic, or the DDoS, shows up.

• You can't accept the trust trade. The CDN terminates your TLS, which means it can read your traffic. For most sites that's the business model, and it's fine. For a handful (dissident journalism, certain health data) that trust isn't acceptable, and TLS stays end to end.

Most public sites answer yes three times, which is why "just put a CDN in front of it" is such common advice. But common isn't universal. The teams that get burned are the ones who never asked the questions.

Section 9 — Going deep: the principal-level toolkit

So far the edge has been a cache with a good address. Now it becomes a computer.

Edge compute: your code, in 300 cities. Once the CDN has servers everywhere and every request passes through them, the next step is obvious: run your logic there. Cloudflare Workers, Fastly Compute, AWS's CloudFront Functions and Lambda@Edge. You deploy a small function, and it executes at the PoP nearest the user, before the request ever heads for the origin. What runs well there:

• Auth at the edge. Verify the session token (a JWT, a signed token the edge can validate without calling anyone), reject the unauthenticated, in Frankfurt, in a couple of milliseconds, without waking the origin. The origin only sees requests that already proved themselves.

• A/B tests and feature flags. Assign the user to a cohort at the edge, serve the right variant, log the assignment. No origin round trip to decide which button color to show.

• Personalization assembly. Fetch the static page shell from the edge cache, fetch the user's name and cart from the origin, stitch them together at the PoP. A personal page at edge latency.

• Request shaping. Normalize URLs, block bad bots, add security headers, redirect old paths: the janitorial work that used to cost an origin round trip each.

The shape of the future: the worker authenticates, personalizes, assembles, and the origin degrades into a fragment API. The mental shift is that the edge isn't the cache in front of your app anymore; it's the outer layer of your app.

The constraints are real, and they differ enough by product that "edge compute" is really four different things:

Two things follow. First, "under a millisecond" is true of CloudFront Functions and not of the others; a Worker or a Lambda@Edge function costs single-digit to tens of milliseconds, which is still a bargain against a 100 ms origin round trip but isn't free. Second, design as if you have 10 ms of CPU and no local state, whatever the plan says, because that keeps workers small, stateless, and boring. Auth, routing, assembly: yes. Business logic with twelve database calls: no. The AWS pair also splits by when the function runs: viewer-side (every request, before the cache) versus origin-side (only on misses), and picking the wrong trigger is the difference between a function that runs a million times and one that runs a thousand.

State at the edge is the part that needs the most care. There is no long-lived state inside a worker; each invocation starts empty. What the platforms offer instead is a global key-value store (Cloudflare KV, Fastly's KV store) that's eventually consistent and built for read-heavy data like feature flags and redirect maps, where a write in Virginia is visible in Tokyo up to a minute or more later, and, for the cases that truly need one authoritative copy (a counter, a rate limit, a collaborative session), a single-location coordination primitive like Cloudflare's Durable Objects, which gives you strong consistency by giving up the "everywhere" property. Know which one your use case needs before you write the worker; a rate limit on eventually-consistent KV isn't a rate limit.

And debugging distributed edge code is its own skill, which deserves more than a warning: run the worker locally in the vendor's emulator before deploying, tail its logs live from the CLI (wrangler tail on Cloudflare) while you reproduce the problem, ship the edge logs to the same place as the rest of your telemetry so a request can be followed from the PoP to the origin, and put a request ID on every edge-generated response. The observability post (#9) covers the rest.

Consistent hashing for cache routing. Inside the CDN, and inside your own multi-node caches, keys have to be assigned to servers so that adding or removing a server doesn't reshuffle everything. The answer is consistent hashing: hash the keys and the servers onto the same ring, and each key belongs to the next server clockwise. A server joins: it takes over only the keys between its predecessor and itself, about 1/N of the total on average. A server dies: its keys slide to the next one. The ring localizes the damage of topology changes, which is exactly what you want when PoPs come and go.

Joining takes a slice from one neighbor; dying hands its slice to the next. Compare with naive hash(key) % N: adding a server reshuffles nearly every key, and every reshuffled key is a cache miss, a self-inflicted avalanche. In practice each server is placed on the ring at many points (virtual nodes), so that the slices are even and a dead server's load spreads across everyone instead of landing on one unlucky neighbor. Any cache cluster that changes size wants the ring. The caching post's Section 9 covered the same structure for client-side hashing and the sharding post (#3) has the full treatment; the CDN runs it at continental scale.

Stale-while-revalidate at the edge. The caching post (#1) introduced this (RFC 5861); at the edge it's load-bearing. The edge serves the slightly stale copy immediately, and kicks off a background request to revalidate with the origin. Users never wait on an expiry; the origin sees a trickle instead of a herd, because the CDN's request coalescing (Section 7) collapses the background refreshes to one per cache tier (the RFC itself doesn't promise single-flight; the CDN's coalescing is what delivers it). Two dials: max-age (serve fresh for this long) and stale-while-revalidate (serve stale-but-acceptable for this much longer, refreshing underneath). A news homepage at max-age=60, stale-while-revalidate=300: never slower than the edge, never staler than six minutes, and with tiering, the origin sees about one request a minute rather than one per PoP. One standards gotcha worth knowing: RFC 9111 says s-maxage implies proxy-revalidate, so a compliant shared cache (Cloudflare included) won't serve stale past s-maxage without checking with the origin; if you want stale-while-revalidate at the CDN, pair it with max-age (or the vendor's own edge TTL setting) rather than s-maxage. When someone asks how to make purges less scary, this is half the answer; the origin shield is the other half.

Hit-ratio economics, with the real prices. A CDN bill has four parts: bandwidth (per gigabyte served, tiered by volume and by region), requests (per 10,000), origin-side charges (shield requests, origin fetches), and features (WAF, workers, image optimization). Two vendor models exist: per-gigabyte (CloudFront's pay-as-you-go, Fastly, Akamai) and flat-rate plans with no per-gigabyte line at all (Cloudflare, and since 2025 CloudFront's own monthly plans). The arithmetic for the per-gigabyte model, done properly: raw egress from a cloud origin costs about \(0.09 per GB at the first tier and drops with volume; the CDN's first-tier price is nearly the same (\)0.085 on CloudFront), so at small volumes the CDN saves you almost nothing per byte. The savings come from two other places. Transfer from your origin to the CDN is free, so at a 95% hit ratio the origin pays egress on 5% of the bytes. And the CDN's volume tiers fall faster than raw egress does, blending to roughly $0.03 per GB at petabyte scale. Add request fees ($0.01 per 10,000 HTTPS requests, so a billion requests a month is $1,000, which dominates the bill for small-object traffic), regional multipliers (bandwidth in India and South America runs anywhere from 30% to well over 100% above the US rate, depending on the vendor), and the shield's per-request charge, and model it with your actual hit ratio and your actual byte mix, not the brochure's. Run a week of production traffic through the trial and read the numbers.

Multi-CDN: don't marry the edge. Two CDNs (say Cloudflare plus Fastly, or CloudFront plus Akamai) with DNS steering between them buys three things: negotiation leverage (bandwidth is a commodity; two vendors keep pricing honest), resilience (every CDN has its bad day; you steer traffic to the other in minutes), and performance arbitrage (one vendor wins Asia, the other wins South America; send each region to its winner). The costs are real: two configurations to keep in sync (purge both, or one serves stale), two bills, and cache fragmentation, since each CDN builds its own hot set and your effective hit ratio per CDN drops. Start with one CDN done well. Add the second when the traffic, or the outage, justifies the complexity. The irony a principal will notice: multi-CDN is the resilience post's (#2) "no single point of failure" applied to the thing you bought for resilience.

The operational reality: DNS steers, the deploy pipeline purges both CDNs (forget one and it serves stale for a TTL), and there's a question the diagram forces you to answer: what protects the origin from two CDNs' worth of misses? No vendor's origin shield is shared across CDNs, so if you want a single funnel you run your own, a small Varnish or nginx cache tier in front of the origin that both CDNs fetch through. Without it, each CDN's shield collapses its own misses, and the origin sees two requests per cold key instead of one, which is usually fine and worth knowing.

The Copy at the Doorstep, distilled

For the skimmers and the revisitors: everything above, on one page.

Key numbers:

Every trade-off, in one table:

Three ideas to take with you:

1. Distance is the latency. Code optimization shaves milliseconds; moving the bytes closer shaves hundreds. The CDN isn't a faster server but a different geometry, and geometry beats engineering.

2. The edge is a copy with an expiry date, not a deployment target. Versioned filenames, short TTLs with stale-while-revalidate for the shell, and headers you actually understand are how you stay sane. The teams with fast purges are fine; the teams who rarely need purges are better.

3. The front door becomes the system. The CDN started as a cache and grew into TLS termination, DDoS absorption, geography, and now your code. Design for the edge as the outer layer of your application, because whether you plan it or not, that's what it is.

Further reading

• Cloudflare: What is a CDN?. The clearest walkthrough of edge caching and the request path behind Sections 1–2.

• Cloudflare: Tiered Cache and AWS: CloudFront Origin Shield. How two vendors actually shape the tiers in Section 7.

• AWS: CloudFront invalidation. Purging in production terms, with the limits and charges Section 4 alludes to.

• Cloudflare: Cache-Control behavior. One vendor's precise reading of every directive in Section 4's table, including the s-maxage / stale-while-revalidate interaction.

• RFC 9111: HTTP Caching and RFC 5861: Cache-Control Extensions for Stale Content. The standards behind the directives; 5861 is short and readable.

• James Kettle: Practical Web Cache Poisoning. The research behind Section 8's poisoning story.

• Cloudflare Workers: Limits and AWS: Choosing between CloudFront Functions and Lambda@Edge. The constraints in Section 9's table, from the vendors.

• Martin Fowler: Two Hard Things. The famous line about cache invalidation, with its sourcing; Section 4's joke.

Where you'll meet this

This post is caching at planetary scale: the caching post's (#1) desk and filing room, except the desk is in 300 cities and the filing room is an ocean away. Every idea transfers. Hit ratio is the same number, TTLs are the same dial, stampedes are the same herd, and the origin shield is request coalescing with a passport. It's the resilience post's (#2) quiet ally: anycast dilution absorbs DDoS attacks before they become your outage, and stale-if-error is graceful degradation with better marketing. The consistent-hashing ring is the sharding post's (#3). Signed URLs for private content and the "hide the origin" rule are the security post's (#12), and the origin failover is the load balancer post's (#11) health checks done by someone else. In the URL shortener design, the edge is the unbuilt extension: redirects are the perfect edge workload (tiny, identical, cacheable), and moving them to PoPs is the obvious Step 12. Next up is rate limiting (#8), for when the flood isn't fans but abuse.

Keep exploring: the Core Concepts series

Every post in the series stands alone. Read them in any order.

• From Laptop to Billions of Clicks: Designing a URL Shortener in 11 Steps — the anchor: one design, every concept under load.

• #1 The 100:1 Superpower: Caching — the fastest request is the one you never make.

• #2 The Blast Radius: Surviving the Day Your Dependencies Fail — staying up when everything you depend on goes down.

• #3 Divide and Conquer: Sharding and Partitioning — splitting one database into many without losing your mind.

• #4 The Snowflake Problem: Unique IDs at Scale — naming things when millions are born every second.

• #5 Copies of the Truth: Replication — keeping copies of your data that actually agree.

• #6 Do No Harm Twice: Idempotency — making "just retry it" safe.

• #7 The Copy at the Doorstep: CDNs and Edge Computing — serving from next door instead of across the ocean. (this post)

• #8 The Bouncer's Math: Rate Limiting — saying no politely, at scale.

• #9 What Broke at 3 AM: Observability, p99, and Useful Alerts — knowing what's wrong before your users tell you.

• #10 Do It Later, On Purpose: Async Processing and Queues — the work the user doesn't have to wait for.

• #11 The Traffic Cop: Load Balancing — the box in every diagram nobody explains.

• #12 Assume They're Already Knocking: Security and Abuse at Scale — designing for the users who are designing against you.

• #13 Do the Math First: Estimation for System Design — the two minutes of arithmetic that choose the architecture.

• #14 The Whiteboard Playbook: Taking On Any System Design Challenge — the whole series in forty-five minutes.

Blueprints of Scale — Core Concepts #7. If this helped, the best thanks is a share with someone who's learning.

The problem it solves is easy to state. The moment you have more than one server, someone has to decide which server gets each request. One server melts under real traffic (Step 0 covered that). Three servers share the load nicely, but only if the sharing is somebody's job, and it turns out clients can't do it fairly and DNS can't do it intelligently. So you put a dedicated machine at the front door whose only job is to distribute. That's the load balancer, and it's one of the oldest ideas in distributed systems. It's also one of the least understood, because it looks trivial right up until the day it decides whether your outage is one server or all of them.

We'll cover why one server is never enough and what replaces it; the distribution algorithms (round-robin, least connections, IP hash, power of two choices, consistent hashing); the Layer 4 versus Layer 7 split and the things that come with it, like who the backend thinks it's talking to and where TLS gets decrypted; health checks, the trap inside "deep" health checks, and graceful draining; sticky sessions and why you should try not to need them; what happens when the balancer itself dies; the ways a balancer can make an outage worse; the balancer as the place you deploy from and shed load at; and, at the end, the deeper toolkit: latency-aware routing, client-side balancing, service meshes, subsetting, and how to size the tier itself.

If you're new to this, Sections 1 and 2 need nothing but curiosity. Sections 3 through 8 are the machinery you'll actually operate. Section 9 is for when you're the one who has to size the thing and defend the design. There's a one-page summary at the end under The traffic cop, distilled, and every diagram is also described in the text around it.

Section 1 — One box isn't enough

Saturday, 2 PM. A small API on a single server, doing fine at 200 requests a second. Then a newsletter goes out, traffic goes up tenfold in an hour, and the server does the only thing an overloaded server can do: it gets slower, then slower, then it stops answering. The first fix anyone reaches for is a bigger server. It works, for three months. Then the newsletter goes out again.

The sharding post (#3) covered the economics: doubling a server's power more than doubles its price, and at some point the bigger box doesn't exist. The durable fix is horizontal. Run three servers instead of one, then ten, then fifty. But now the client has to pick a server, and clients are bad at it.

Hand a client three IP addresses and watch what happens. Some clients take the first one and never try the others. DNS round-robin rotates the list, but DNS answers get cached by the client's operating system, by the ISP's resolver, by everything in between, so the rotation is more of a suggestion than a mechanism (each answer carries a TTL, a time to live, and nothing forces a client to re-ask before it expires). One server ends up with 70% of the traffic while the other two idle. And when a server dies, nothing tells the clients. They keep sending requests to the dead address until a human edits DNS and then waits for every cache in the world to expire. (To be fair to DNS: a health-checked, low-TTL DNS service like Route 53 is a legitimate tool for steering traffic between whole regions, and Section 6 comes back to it. It's just the wrong tool for picking between servers a millisecond apart.)

So you put one stable address in front and a dedicated box behind it. Clients talk to the address; the box, the load balancer, talks to the servers. It knows all three of them (or all thirty), it knows which ones are alive, and for every incoming request it picks one. From the client's side there's a single endpoint that never gets tired. From the servers' side the work arrives evenly split.

That's the idea in one line: one address in, N servers behind it, and the distribution is someone's job. The job has three parts, and they're the next three sections: deciding which server gets each request, knowing which servers are alive, and understanding enough of the request to decide well. Everything after that is what happens when the balancer decides badly, can't see, or dies.

A quick map of what "a load balancer" can physically be, because the vocabulary trips people up. It can be a hardware appliance (F5 BIG-IP, NetScaler, the boxes with the big price tags), a piece of software on an ordinary machine (HAProxy, NGINX, Envoy, Traefik), a cloud-managed service (AWS ALB and NLB, Google Cloud Load Balancing, Azure Load Balancer and Application Gateway), or something living inside a cluster (Kubernetes Services via kube-proxy or IPVS, Ingress controllers, the Gateway API). They all do the job this post describes. The words differ by vendor: the set of servers behind a balancer is a pool, an upstream, a target group, or a backend service depending on whose documentation you're reading, and they all mean the same thing.

Section 2 — The doorman's playbook

A load balancer makes one decision, over and over, millions of times a second: this request goes to that server. There are half a dozen standard ways to make it, and each one is a different guess about what "fair" means for your traffic.

Round-robin is the default in nearly every product: request 1 to server A, 2 to B, 3 to C, 4 back to A. No memory, no state, nothing to tune. It's fair as long as every request costs about the same, which they never do. One request is a 2 ms cache lookup and the next is a 900 ms report. Round-robin deals them out evenly and the servers end up unevenly loaded anyway, the same way dealing cards evenly says nothing about who got the aces.

Weighted round-robin fixes one specific unfairness, which is that the servers aren't identical. The new 64-core box gets weight 4, the old 16-core box gets weight 1, and the rotation sends four requests to the big one for every one to the small one. It's simple and static, and it's wrong the moment the weights stop matching reality. Section 7 has more to say about that.

Least connections stops counting requests and starts looking at the servers. Each new request goes to whichever server has the fewest active connections. The one still chewing on the 900 ms report has 40 connections open; the one that's idle has 3; the next request goes to the idle one. For uneven work (long-lived connections, websockets, a mix of cheap and expensive endpoints) it's usually the right default, and it's what I'd start with unless I had a specific reason not to. One caveat for later: when you have many balancers in front of the same fleet, each with its own view of connection counts, they can all pick the same "emptiest" server at the same moment and pile onto it. Section 9 has the fix.

Least response time goes a step further. The balancer tracks how slow each server has been recently, usually as an exponentially weighted moving average (EWMA: a running average that counts the last few seconds more heavily than the last few minutes), and prefers the fast ones. This catches the server that's alive but struggling: the one stuck in garbage-collection pauses (when a managed runtime like the JVM stops the world to reclaim memory), or sharing a physical host with a "noisy neighbor" that's eating the CPU. The risk is a feedback loop: the fast server gets more traffic because it's fast, until it isn't. It needs damping or it oscillates.

IP hash is the cheap way to get stickiness: hash the client's IP, take hash(ip) mod N, and the same client always lands on the same server. No cookies and no state in the balancer. It has two problems. Adding one server reshuffles the modulo for everyone (the sharding post solved that exact problem with consistent hashing, and most balancers offer it as a mode). And an IP is not a person: behind a mobile carrier's or a corporate network's NAT (network address translation, which hides many devices behind one public address), one address can be thousands of users, and all of them land on the same server.

Power of two choices is my favorite, because it seems too simple to work. Pick two servers at random and send the request to the less loaded of the pair. Michael Mitzenmacher's analysis (his 1996 thesis, published in IEEE TPDS in 2001, building on Azar, Broder, Karlin and Upfal's 1994 balls-in-bins result) showed this gives you exponentially better balance than picking one at random, close to what you'd get from checking every server, for the cost of checking two. It's the algorithm for when you want least-connections behavior without the balancer having to keep a global, always-accurate picture of every server's load. HAProxy's balance random does two draws by default for exactly this reason.

The last row needs a sentence. When each server holds its own cache of hot data, you want the same request to hit the same server every time; otherwise every server ends up caching everything and the hit ratio collapses. Consistent hashing (the ring and virtual nodes from the sharding post) gives you that affinity and keeps most of it intact when servers come and go. The CDN post (#7) used it at the edge for the same reason.

The failure I've seen most often here is round-robin in front of a mixed workload: 2 ms lookups and 30-second file conversions sharing one fleet. The conversions land wherever the rotation happens to put them, those servers build up minutes of queue, and the cheap lookups get stuck behind them. Switching to least connections fixes it in an afternoon. The algorithm is a bet about your workload, so bet on what your traffic actually looks like rather than on whatever the default was.

Section 3 — Does the doorman read the letter?

Everything in Section 2 assumed the balancer picks from one pool of interchangeable servers. The next question is how much of the request the balancer can actually see, and the answer splits balancers into two families. The names come from the OSI networking model, where layer 4 is the transport layer (TCP and UDP: addresses and ports) and layer 7 is the application layer (HTTP: URLs, headers, cookies).

A Layer 4 balancer reads the envelope: source IP, destination IP, port. It never looks inside. A TCP connection arrives, the balancer picks a backend, and the packets go through; the backend terminates the connection, does the TLS handshake, and reads the HTTP. Because it never opens the letter, an L4 balancer is fast, cheap, and indifferent to protocol. It balances HTTP, websockets, raw TCP, your custom binary protocol, whatever you have. It also can't make decisions based on what's in the letter. It can't send /api to one fleet and /static to another, because it never sees the URL. (It can route by port, and a TLS-aware L4 balancer can peek at the server name in the handshake, but that's about the limit.)

A Layer 7 balancer reads the letter. It terminates the TCP connection and the TLS session itself, parses the HTTP request, and routes on anything inside it: the Host header (api. to the API fleet, www. to the web fleet), the path (/images to the image servers), a custom header (X-Tenant: acme to that tenant's fleet), a cookie (that's stickiness, Section 5). Then it opens a new connection to the backend it chose. This is where routing gets expressive, and where the costs come from.

The trade:

• L4 is faster and cheaper per request. No decryption, no HTTP parsing, a smaller CPU bill, and it works for anything on TCP or UDP. But it's limited to routing by address and port, with no content routing, no cookie stickiness, and no header-based canaries.
• L7 is smarter and heavier. It terminates TLS (CPU cost, plus the certificates now live on the balancer), parses every request, and adds a small hop of latency. In exchange you get path, host, and header routing, cookie affinity, request rewriting, per-route policies, and a place to hang a web application firewall (a WAF, which inspects requests for known attack patterns before they reach your code).

The rule of thumb: L4 when the backends are interchangeable or the protocol isn't HTTP; L7 when routing depends on what's in the request. Most real systems run both, with a cheap L4 tier at the very edge spreading traffic across a fleet of L7 balancers that do the clever part. The CDN post's edge is layered the same way, TCP at the bottom and HTTP intelligence on top.

Now three things that fall out of this split and cause more tickets than the algorithms in Section 2 ever will.

How the packets actually get to the backend. There are three shapes. A full proxy terminates the client's connection and opens a second one to the backend; every L7 balancer is a full proxy, and so are HAProxy's and NGINX's TCP modes. A NAT balancer rewrites the destination address on each inbound packet and rewrites the source on the way back, so both directions flow through it. And direct server return (DSR, sometimes done with tunneling) has the balancer touch only the inbound packets: the backend answers the client directly, and the response never passes through the balancer at all. DSR is what you want when responses are far bigger than requests, which is most of the web, because the balancer only has to carry the small half. It's how the big stateless L4 tiers in Section 6 work. The point for now is that "Layer 4" covers all three, and which one you're running changes what the backend sees.

Who the backend thinks it's talking to. With a full proxy, every connection arriving at a backend comes from the balancer's IP. Your access logs show one address. Your per-IP rate limits (post #8) throttle the balancer. Your geo-lookup says everyone is in the same rack. The fix at L7 is a header: the balancer appends the real client address to X-Forwarded-For (or the standardized Forwarded header from RFC 7239). At L4, where there are no headers to add, the PROXY protocol prepends a small line with the original addresses to the start of the TCP stream, and the backend has to be configured to expect it. One rule to tattoo somewhere: trust only the address your own balancer appended. X-Forwarded-For is a plain header that any client can send with any value, and a rate limiter that believes the first entry is a rate limiter an attacker can point at somebody else. The security post (#12) comes back to this.

Where TLS ends. An L7 balancer holds your private keys and decrypts every request, which means decrypted traffic now travels your internal network. That's a security boundary and a compliance question, and it's the kind of thing to raise in the design review rather than discover in an audit. You have three options. Termination: decrypt at the balancer, plaintext to the backends; simplest, fine inside a trusted network. Re-encryption: decrypt at the balancer to route, then open a fresh TLS connection to the backend; this is what service meshes do, often with both sides presenting certificates. Passthrough: never decrypt; the balancer reads only the server name from the TLS handshake (SNI) to pick a pool, and the backend holds the keys. Passthrough keeps keys off the balancer but gives up everything in the L7 column. Two operational notes for whichever you pick: certificate renewal should be automated (ACME, the protocol behind Let's Encrypt), because expired certificates are an outage class of their own, and TLS session resumption at the balancer is what keeps the handshake CPU cost affordable.

One more, because it surprises people who've moved to HTTP/2 or gRPC. HTTP/2 multiplexes many requests over one long-lived TCP connection. A connection-level balancer (L4, or an L7 balancer that isn't HTTP/2-aware on the backend side) picks a backend once, when the connection opens, and every request on that connection goes to the same place for as long as it lives. Balance that looks perfect by connection count can be terrible by request count, and a new backend added to the pool gets nothing until clients open new connections. The fixes are to balance per request at L7, to have backends set a maximum connection age and send a GOAWAY (HTTP/2's polite "please reconnect" frame) so clients reconnect and re-spread, or to balance on the client side (Section 9). HTTP/3 over QUIC (the newer transport that runs over UDP) has its own version of this: a QUIC connection can survive the client changing IP addresses, so an L4 balancer has to route by the QUIC connection ID rather than the address tuple, or migrations break.

Section 4 — Dead servers and graceful goodbyes

A deploy rolls out and server 7's new binary has a bug: it accepts connections and then hangs. The balancer keeps sending it a third of the traffic, because as far as the balancer can tell, server 7 is fine. The TCP handshake succeeds. Users start timing out. The deploy dashboard says success. The service is down.

The balancer only knows what it checks, and health checks are how it checks. There are two kinds.

Active checks are the balancer probing on its own schedule: every few seconds it sends each server a request, typically an HTTP GET to /health, and after some number of consecutive failures the server is pulled from the pool. The defaults vary more than you'd expect. HAProxy probes every 2 seconds and ejects after 3 failures; Google's cloud balancer every 5 seconds, 2 failures; AWS's ALB every 30 seconds, 2 failures; Kubernetes every 10 seconds, 3 failures. None of those numbers is magic. The interval sets how long a dead server keeps receiving traffic before anyone notices, and the failure count sets how much noise you tolerate before acting.

Passive checks are the balancer watching real traffic. A backend starts timing out, returning 500s, resetting connections; the balancer notices without sending a single probe and pulls it. Passive checking costs nothing extra and reacts to actual user pain, but by definition it only fires after someone has already felt that pain. (Open-source NGINX only does passive checks, and its default of max_fails=1 means one failed connection or timeout ejects a server for ten seconds (HTTP 5xx responses count only if you list them in proxy_next_upstream). Worth knowing before you wonder why servers keep blinking out of the pool.)

The interval matters less than two other things: what the probe actually tests, and how the thresholds are shaped.

What the probe tests, and the trap on both sides. The shallow failure is obvious: /health returns 200 because the web server process is up, while the database connection pool behind it is exhausted and every real request fails. A check that doesn't exercise anything is a lie you tell the balancer. So the instinct is to make the check deep: hit the database, hit the cache, exercise the real request path. And that instinct has a trap of its own, which is easy to miss until it takes down a fleet. If every server's health check touches the same shared database, and that database blips for five seconds, every server fails its check at the same moment. The balancer, doing exactly what it was told, ejects all of them. A five-second database hiccup that the servers could have ridden out (serving cached data, returning degraded responses, or just retrying) becomes a total outage, followed by every server rejoining at once when the database recovers. A dependency check that fails identically on every server isn't telling the balancer which server to avoid; it's telling it the whole system is sick, and that's an alert (post #9), not a routing decision.

The advice that survives both traps: the health check should verify this server's ability to do its job, meaning the things that differ between servers, like its own connection pool, its own disk, its own configuration, and whether it finished starting up. Shared-dependency status belongs in a separate "degraded" signal that gets reported but doesn't fail the check. And the balancer needs a fail-open rule for the case where everything looks unhealthy anyway: Envoy calls this the panic threshold, and by default, if fewer than 50% of a pool's hosts are healthy, it ignores health entirely and sends to all of them, on the theory that a fleet that's all "unhealthy" is more likely a broken check or a shared dependency than a fleet that's all dead. Amazon's Builders' Library article on health checks is the best long-form treatment of this balance I know of.

If you run on Kubernetes, the same idea comes packaged as two probes with different consequences. A liveness probe failing means "restart this container." A readiness probe failing means "stop sending this pod traffic, but leave it running." The balancer consumes readiness. Confusing them is common and expensive: a liveness probe that checks the database restarts every pod in a crash loop when the database is down, which is the fleet-wide trap above with extra steps. There's a startup probe for slow-booting services so liveness doesn't kill them mid-boot, and there's a race at shutdown worth knowing: a pod can receive its termination signal before the endpoint removal has propagated to every proxy, so requests still arrive at a container that's already exiting. The standard workaround is a preStop hook that sleeps a few seconds before the process gets its signal.

Flapping and hysteresis. A server that oscillates (fail, pass, fail, pass) gets yanked in and out of the pool, and every yank reshuffles connections onto the survivors. The fix is hysteresis: different bars for leaving and rejoining. Three failures to get pulled; ten consecutive clean probes, or a full minute of them, to get back in. Leaving is easy, returning has to be earned. Note that several products' defaults do the opposite (HAProxy needs 3 failures to eject and only 2 successes to rejoin; Kubernetes 3 and 1), while AWS's ALB (2 to eject, 5 to rejoin) gets it right, so this is a setting you make deliberately. And once a server is back, ramp its traffic up gradually rather than handing it the full share at once. That's slow start, in Section 7.

Then there's the planned version of death. Every deploy, every scale-down, every machine retirement takes a server out of the pool on purpose, and the civilized way to do it is connection draining: the balancer stops sending new requests to the retiring server, waits for the in-flight ones to finish, and only then lets it shut down. The waiting period is a setting: Kubernetes gives a pod 30 seconds by default, and AWS's ALB waits 300 seconds by default (its "deregistration delay"). Skip the draining and every deploy drops whatever was mid-request. That's the little spike of 500s that shows up in deploy dashboards, which teams first learn to recognize and then, eventually, learn to stop tolerating.

Draining has details that bite. For HTTP/1.1 keep-alive connections, the server should answer with Connection: close so the client reconnects elsewhere. For HTTP/2 it sends GOAWAY. A websocket never "finishes" on its own, so a drain needs a deadline after which the server closes the connection deliberately and the client has to know how to reconnect, ideally with some random jitter so ten thousand reconnects don't land in the same second. And the application has to handle its termination signal (SIGTERM) by finishing work rather than exiting immediately, or the balancer's patience is wasted.

Two more items that live here and generate a remarkable number of tickets:

• Timeout mismatch and the intermittent 502. An L7 balancer keeps a pool of idle connections to each backend and reuses them. If the backend's keep-alive timeout is shorter than the balancer's idle timeout, the backend closes an idle connection at the exact moment the balancer picks it for a new request, and the client gets a 502 Bad Gateway for no reason anyone can reproduce. The rule is that the backend's idle timeout must be longer than the balancer's. The canonical pairing is an ALB at 60 seconds in front of NGINX at 75. There's a related layering for request timeouts: if the balancer gives up at 30 seconds while the backend is happy to work for 60, the balancer returns a 504 and the backend keeps working on a response nobody will read.
• Health checks are traffic. N balancers each probing M backends every few seconds is N × M / interval requests per second of overhead. It's small until it isn't, and a health endpoint that does real work (the deep check above) multiplies it. Managed balancers probe from several nodes and vote, which is good for accuracy and worth remembering when you count.

The balancer's picture of the world is only as good as its probes. Shallow checks lie about the server. Shared-dependency checks lie about the fleet. Symmetric thresholds make servers flap. Deploys that don't drain drop requests.

Section 5 — The sticky problem

The best session is the one the server never keeps.

Some applications keep per-user state on the server: a shopping cart in memory, a half-completed form, a websocket connection with its list of subscriptions. The moment that state lives on server 3, the balancer loses its freedom. User A's next request has to go to server 3 or the cart is gone. The mechanism for guaranteeing that is sticky sessions, or session affinity: the balancer sets a cookie on the first response (SERVERID=3), the client sends it back with every request, and the balancer routes by it. Same user, same server, cart intact. (Balancers can insert their own cookie for this, or key on one your application already sets; the application cookie is nicer when you want the affinity to expire with the login.)

It works, and it costs you three things, slowly.

• Distribution rots. Stickiness overrides every algorithm in Section 2. The heaviest sessions pile up on whichever servers they happened to land on, and your carefully chosen least-connections becomes decorative.
• Failover loses state. Server 3 dies and every session pinned to it evaporates: users logged out, carts emptied, forms reset, at exactly the moment the system is already having a bad day.
• Scaling reshuffles. Adding servers, removing servers, deploying: every topology change re-pins some fraction of users, and never evenly.

There's a fourth cost hiding in the balancer itself. If the balancer keeps the session-to-server mapping in its own memory (HAProxy calls these stick tables), that mapping has to be replicated to its standby partner or a balancer failover re-pins everyone. Cookie-carried routing sidesteps this because the mapping travels with the client.

The fix is the one the caching post (#1) already made: move the state out of the server. Sessions go in Redis or the database, every server becomes interchangeable, and the balancer gets its freedom back. The cart survives server 3's death because the cart never lived on server 3. Websockets are the real exception, since a persistent connection is pinned by physics. There you pin deliberately, by IP hash or cookie, and design the reconnect path so the client can reconnect to any server, resubscribe, and have that server rebuild its subscription state from the shared store.

Sticky sessions aren't wrong, exactly; they're a loan. They let you ship stateful servers today and pay in uneven distribution, failover pain, and scaling friction for as long as the system lives. If a design needs stickiness, the first question in review should be what state, and why it can't live outside the server. Usually it can.

Section 6 — The doorman is a box too

The diagram hides something. The load balancer is a server, and servers die. When the box whose job is distributing traffic dies, all the traffic stops, not one-Nth of it. The thing you built to eliminate the single point of failure is now the single point of failure, and every serious load-balancing design starts by admitting it.

The fixes come in layers.

HA pairs. Two balancers, one address. The usual setup is active-passive: the primary holds a floating virtual IP (a VIP: an address that isn't tied to one machine and can be claimed by whichever box is in charge), the standby watches the primary's heartbeat, and when the heartbeat stops the standby claims the IP. On Linux this is usually VRRP (a protocol for passing a shared address between machines) via keepalived, and failover takes a few seconds. In-flight connections drop unless the pair mirrors connection state, which the hardware appliances and some software setups do; new connections flow immediately either way. Active-active is the more capable version. Both balancers serve, with the network splitting connections across them (ECMP, equal-cost multi-path routing, hashes each connection to one of several equally good next hops), and either can die without anyone noticing. It's the replication post's (#5) lesson again: the only real fix for a single box is not needing that box.

Scaling the tier. One HA pair handles a lot, but not an unlimited amount, and the tier scales the same way the app tier did: more balancers, with something in front distributing to them. That something is usually the network itself, through ECMP inside your own network and anycast across the internet. Anycast means the same IP address is announced from many locations, and BGP (the routing protocol the internet's networks use to tell each other which addresses they can reach) delivers each client to the nearest one. The CDN post (#7) took anycast apart in detail; the mechanism and the reason are the same here.

There's a catch in "let ECMP spread connections across N balancers," and the hyperscalers all hit it. ECMP hashes each packet's addresses to pick a balancer. Add or remove a balancer and the hash function's output changes for a large fraction of flows, so packets from an existing connection start arriving at a balancer that has never seen it, which resets the connection. The fix is a stateless L4 tier where every balancer uses the same consistent hash to pick the backend, plus a local table of flows it has seen, so any balancer receiving any packet routes it to the same backend. Google's Maglev (published 2016), Meta's Katran (built on XDP/eBPF, which lets a program run inside the Linux kernel's network path), Cloudflare's Unimog, and GitHub's GLB are all versions of this idea, and they typically use DSR from Section 3 so the balancers only carry the inbound half. You probably won't build one. It helps to know why they exist when the cloud balancer's documentation starts talking about connection draining on scale-in.

Global load balancing is the same idea at planetary scale, with two mechanisms and different trade-offs:

• DNS-based. A geo-aware DNS server answers "where is api.example.com?" with the address of the nearest healthy region: Virginia for New York clients, Frankfurt for Berlin. It's simple and needs no special network. But DNS answers get cached (Section 1's problem again, now worldwide), so failover means waiting for TTLs to expire across every resolver on earth. Minutes, not seconds, and some clients pin the first IP they got and never re-resolve at all. And "nearest" is approximate, because DNS sees where the resolver is, not where the client is; the EDNS Client Subnet extension (RFC 7871) lets resolvers pass along part of the client's address to fix this, and some of the big public resolvers do (Google's and OpenDNS's) while the privacy-focused ones don't (Cloudflare's 1.1.1.1, and Quad9 by default).
• Anycast. One IP, announced from every region, and BGP does the rest. Failover is a routing change, so there are no caches to wait out. Inside your own network that converges in seconds; on the public internet, tens of seconds to a couple of minutes, and a route change can break long-lived TCP connections mid-flight. This used to require your own address space and BGP relationships, which kept it a big-operator tool. It's now something you can buy: AWS Global Accelerator, Google's global load balancer, and Cloudflare all put anycast in front of ordinary origins.

Geo-routing, sending EU users to EU servers, is where this meets the CDN post's edge: same anycast, same points of presence, same physics. And the same caveat: geography is a routing input, not a guarantee. BGP decides "nearest" by path cost rather than kilometers, and paths change.

One level down from regions, cloud balancers are zonal. A cloud balancer is a fleet of nodes spread across the availability zones you enabled, and the Layer 4 kind (AWS's NLB) by default sends traffic only to backends in the same zone as the node that received it; cross-zone balancing evens things out at the cost of cross-zone bandwidth charges and a millisecond or two of latency (the ALB, AWS's Layer 7 balancer, always balances across zones and doesn't charge for it), and the trade shows up in a specific way: if one zone dies, do the survivors have the headroom to absorb its share? Section 7 does that arithmetic.

A question worth asking in any design review: what balances your load balancer? If the answer is "nothing, it's one box," that's fine for a side project and a known risk for anything else. Better to name it than discover it.

Section 7 — When the doorman makes it worse

The balancer did its job perfectly. That was the problem.

Ten servers. One dies at peak, a real hardware failure. The health checks catch it within seconds, pull it, and redistribute its share across the nine survivors. Each survivor now carries about 11% more than it did. They were at 85% utilization; now they're at 94%. Latency climbs, timeouts start, and the passive checks see the timeouts and conclude that a second server is unhealthy. Out it goes. Its load spreads across eight, which puts each of them past 100%. Queues explode. The balancer pulls a third. It is faithfully, correctly, and algorithmically killing the fleet one server at a time, and by the time a person intervenes, "one server died" has become "everything is down." This is cascading failure, and the balancer is what transmits it.

The arithmetic underneath is worth having in your head, because it tells you how much headroom to buy. If you want to survive losing k of N servers, the survivors have to absorb the whole load, so your normal utilization can't be above (N − k) / N. Ten servers at 85% can lose one (survivors at 94%) but not two (survivors at 106%). Ten servers at 70% can lose two and still be under 90%. The resilience post (#2) makes the same point from the other side, and Section 9 of the estimation post (#13) turns it into a purchase order.

Three more ways the balancer can cause the outage it exists to prevent:

Thundering herd on failover. The dead server's traffic doesn't trickle over to the survivors; it lands all at once, mid-peak. The mirror-image problem is a server joining cold: empty caches, an unwarmed JIT (the just-in-time compiler that managed runtimes use to speed up hot code after it has run a while), and a full share of traffic from the first second. The fix for the cold-join problem is slow start. A rejoined or newly added server gets a trickle first, something like 5% of its share ramping to full over a minute, so caches fill and the runtime settles before the flood. Envoy supports it (slow_start_config), HAProxy has slowstart, AWS's ALB has a slow-start setting, and NGINX has had it in open source only since 1.29.6 (March 2026); before that it was a commercial NGINX Plus feature. It's a few lines of configuration and it has saved more fleets than any algorithm choice.

Retry amplification. The client makes up to three attempts. The balancer retries a failed backend attempt on another server, so up to two attempts per client attempt. The server's own database client makes two attempts. One user action becomes 3 × 2 × 2 = 12 database operations, during an incident, when everything is already saturated. (Google's SRE book uses a nastier version of this: three layers each allowing three retries is 4 × 4 × 4 = 64 attempts per user action.) This is the resilience post's retry problem, and the balancer is one of the layers doing the multiplying. The fix is retry budgets at every layer, the balancer included: each layer gets a capped percentage of its traffic as retries rather than a per-request multiplier. Envoy's retry budget, once you enable it, is 20% with a floor of three concurrent retries; Google's guidance is a 10% per-client budget. Two safety rules travel with this: the balancer should only retry on connection failures and on idempotent methods (ones safe to repeat, like GET and PUT; never a blind POST; post #6 is about why), and each attempt needs its own per-try timeout so the retries don't stack into one enormous wait.

Stale weights. The new 64-core boxes were supposed to get weight 4, someone left them at 1, and they idle while the old boxes melt. Or a canary weight of 5% never gets removed, and six months later 5% of production is running an unmaintained build. Weights are configuration, configuration rots, and the balancer obeys rotten configuration perfectly.

The defenses come as a set. Cap how much of the fleet the balancer may eject at once: Envoy's outlier detection defaults to a maximum of 10% of a pool, and the cap is the important part, because a balancer that can fire everybody eventually will. Keep the panic threshold from Section 4, so that a pool that looks entirely unhealthy gets traffic anyway. Slow-start rejoined servers. Budget retries per layer. Add jitter to client reconnects so a balancer failover doesn't produce a synchronized reconnect storm. Audit weights the way you'd audit firewall rules. Underneath all of them is one idea: the balancer is a control loop, and a control loop without damping will oscillate the system to death.

Section 8 — The control plane

Step back and notice what the balancer sees: every request, every response code, every latency, every backend's health. The whole system's vital signs, in one place, in real time. No other component has that view, which makes the balancer more than a distributor. It's the one component positioned to steer. Deploys and disasters both run through it. (People call this the control plane: the part of a system that decides where traffic goes, as opposed to the data plane, which carries it. The balancer straddles both.)

Blue-green and canary deploys are weights. Two fleets, blue running the current version and green running the new one. A blue-green deploy flips the balancer from 100% blue to 100% green in one configuration change; rollback is flipping it back, and the price is that nobody gets gradual exposure. A canary is the careful version, and it's just weighted round-robin on a schedule: 1% to green for an hour while you watch the error rate, then 5%, then 25%, 50%, 100%. The balancer's weights are the deploy tool and its metrics are the safety signal. If green's p99 latency (the latency that 99% of requests come in under, which is where problems show up first) diverges from blue's, the weights go back before most users notice. Two refinements worth knowing: you can target the canary by header or cookie instead of by percentage, so employees see the new version first; and traffic mirroring (Envoy's request mirror policy, NGINX's mirror) copies live requests to the new version and throws away its responses, so you can watch it handle real traffic with zero user exposure. Mirroring only works for requests without side effects, or the new version will place real orders twice. Since the deploy strategy is literally a load-balancing configuration, treat it like one: versioned, reviewed, and rolled back automatically.

Shedding at the front door. The balancer watches backend queue depth, and when it crosses a line, it stops queueing and starts refusing, quickly.

When the backends saturate, somebody has to say no, and the balancer is the best somebody. It sits in front of the overloaded servers, it sees the whole picture, and saying no there costs almost nothing. The mechanisms are simple: cap concurrent connections per backend, so the excess gets a fast 503 instead of a slow timeout; cap the queue, so requests beyond N are rejected immediately (a fast no beats a slow yes that was going to time out anyway); and send a Retry-After header so well-behaved clients back off instead of hammering. The status code matters: 503 means we are overloaded, 429 means you exceeded your limit, and the rate-limiting post (#8) is about the second one. If requests carry a priority or criticality header, the balancer can shed the cheap ones first, which the resilience post calls admission control; this is admission control located at the front door. One way or another the balancer decides how the system behaves under overload. It can do that deliberately, in configuration, or accidentally, in collapse.

The shape this usually takes is a flash sale. Backends saturate, no shedding is configured, and the balancer keeps accepting connections and queueing them, tens of thousands deep, until every client times out and retries, which doubles the queue. The site is technically up (connections accepted!) and completely failing (nothing completes). The fix is one setting: cap the queue at a few hundred per backend and return 503 past that. The next sale is boring. A couple of percent of shoppers see "try again in a moment," everyone else checks out, and that's the whole difference between everyone suffering and a few people retrying.

Since the balancer sees everything, it's also the natural place to stamp each request with an ID and write the access log. A request ID injected at the front door and passed down through every service is what makes a trace possible later (post #9), and the balancer's access log is often the only complete record of what clients actually sent.

Section 9 — Going deep: the principal-level toolkit

You've picked the least-loaded server. Why is it still the slow one?

Because load isn't latency. The server with 3 connections might be in the middle of a four-second garbage-collection pause; the one with 42 might be chewing through trivial cache hits at 2 ms each. Least connections routes around busyness and knows nothing about speed. In a system where the p99 is the promise, speed is what matters.

Latency-aware routing is the answer: route by observed latency rather than connection count. Track each backend's recent response-time distribution (the EWMA from Section 2, or a small reservoir of recent samples) and prefer the backends whose tail is short right now. A server sliding into GC pauses gets quietly avoided until it recovers. No threshold crossed, no ejection, just fewer requests at exactly the moment it needs fewer. The named implementations are mostly client-side: Finagle and Linkerd's "peak EWMA" combined with power of two choices, Envoy's least_request with its active-request bias, NGINX's least_time (open source since 1.31.0). Central balancers mostly don't do this, which is one of the reasons the client-side approaches below exist. A related trick is to have backends report their own load (CPU, queue depth) in response headers or a side channel, since the server knows it's in trouble before the balancer's samples do; Envoy calls this ORCA, and it's what Google's balancers have done for years.

The subtler version is hedged requests, which the resilience post introduced: send the request to one server, and if it hasn't answered within the p95 time, send a duplicate to a second server and take whichever answers first. You're not predicting which server will be slow; you're declining to wait for it. The cost is the duplicated work, which stays bounded because you only hedge the tail (Dean and Barroso's "The Tail at Scale" paper hedges at the 95th percentile, adding about 5% load, and reports a Bigtable case where hedging after a fixed 10 ms cut the 99.9th percentile from 1,800 ms to 74 ms for 2% extra requests). The payoff is a p99 that stops caring about any individual server's bad minute.

Client-side load balancing removes the middleman. Instead of a box at the front door, each client holds the list of servers and picks one per request. This is the gRPC model: no extra hop, no single box to scale, no added latency, with the client library doing the health checking, retries, and weighted choice itself. The prices: every client has to be smart, library upgrades roll out slowly, the service-discovery system that feeds clients the server list becomes a critical dependency, and every client opens connections to every server, which is N × M connections across the system. (A buggy picker in one service only poisons that one service, which is a cost and a benefit at the same time.) It works well inside the system, service to service, where you control the clients. It doesn't work at the edge, where the clients are browsers and phones you don't.

The N × M problem has a standard answer called subsetting, from the Google SRE book: each client connects to a fixed-size subset of the backends rather than all of them, chosen deterministically so that every backend ends up with roughly the same number of clients. Too small a subset and one client can't spread its load; too large and you're back to N × M. The book's deterministic subsetting algorithm is a few dozen lines and worth reading once. This is also where the herding caveat from Section 2 gets resolved: with many independent balancers or clients each picking "the emptiest server," they all pick the same one at the same instant. Power of two choices, or a small random subset per client, breaks the synchronization.

Service meshes are client-side balancing packaged as infrastructure. A sidecar proxy (usually Envoy) sits beside every service instance (in Kubernetes, a second container in the same pod) and does everything Sections 2 through 8 describe: least connections, health checks, outlier ejection, retries with budgets, canary weights, and mutual TLS (mTLS, where both sides of every connection present certificates), all configured centrally and executed locally. What a mesh really does is move the load-balancing logic out of a central box and into every pod. That buys you per-service policy and removes the central bottleneck, at the cost of running and configuring a distributed system to manage your distributed system, plus a proxy hop per call. The newer "proxyless" designs (gRPC clients that speak the mesh's xDS configuration protocol directly, and Istio's sidecar-less ambient mode, which runs one proxy per node plus optional waypoint proxies) are attempts to keep the central policy without the per-pod sidecar tax. My rule: adopt a mesh when you have dozens of services that need different policies. Until then the central balancer is simpler, and the simplicity is the point.

Sizing the tier itself. The balancer is infrastructure with its own ceiling, and someone has to do the math:

• A single software balancer (NGINX, HAProxy, Envoy) on a serious machine pushes tens of gigabits, up to around 100 Gbps with a 100 GbE card, and holds on the order of a million concurrent connections. HAProxy has published 2 million HTTPS requests per second and 92 Gbps on one 64-core cloud instance. The spread is wide because it depends on TLS (termination is the expensive part; passthrough is nearly free) and on how much L7 parsing you do. Treat these as order-of-magnitude numbers and measure your own.
• The binding constraints are usually CPU, because every new connection costs a TLS handshake (session resumption and keep-alives are how you make that affordable), and then a set of limits that surprise people because they aren't about bandwidth at all: the file descriptor limit (every connection is one), the kernel's connection-tracking table if the firewall (netfilter) or address translation (NAT) is in the path (when it fills, new connections drop and the kernel logs nf_conntrack: table full), and for a full proxy, ephemeral ports. A proxy opening connections to a backend has roughly 28,000 to 64,000 source ports per source IP to that backend's address, and a busy proxy can exhaust them; the fixes are keep-alive pools to backends, HTTP/2 to backends, and multiple source IPs.
• When one pair isn't enough, add pairs, front them with ECMP or anycast, and let the network distribute to the distributors. The recursion bottoms out at BGP, which is the internet's own load balancer and scales fine, with the convergence caveats from Section 6.
• Cloud balancers bill in capacity units (an ALB's LCU counts new connections, active connections, bandwidth, and rule evaluations, and charges for the largest), so "how big" is also a line item; the estimation post (#13) is where that arithmetic lives.
• And because the balancer sees every request, it's the cheapest place to measure. Request rates, error rates, latency histograms per backend: export all of it. The component that steers should also be the one that reports.

One last way to look at it. Beginners think the load balancer distributes traffic, but what it actually distributes is fate: which server's bad minute becomes whose bad experience, whether one dead box stays one dead box, whether overload is shared across everyone or shed from a few. Every section of this post was a version of that decision. It deserves to be designed like it matters, because when it fails, it's the only box whose failure is total.

The traffic cop, distilled

For the skimmers and the revisitors: everything above, on one page.

Key numbers:

Every trade-off, in one table:

Three ideas to take with you:

1. The load balancer distributes fate, not traffic. Which server's bad minute becomes whose outage, whether one dead box stays one dead box, whether overload is shared or shed: every section was this decision in a different form.
2. Every control loop needs damping. Hysteresis on health checks, ejection caps and a panic threshold, slow start on rejoin, retry budgets. The balancer is a feedback loop, and undamped feedback loops oscillate systems to death.
3. The balancer is the cheapest place to steer and to see. Deploys, canaries, shedding, request IDs, and the system's vital signs all live at the front door. Instrument it, configure it deliberately, and never let it be one box.

Further reading

• NGINX: HTTP load balancing. The practical reference for weighted round-robin, least connections, IP hash and health checks; note that active health checks are still an NGINX Plus feature (open-source NGINX has only passive max_fails), and slow start only joined open source in 1.29.6. Behind Sections 2, 4, and 7.
• HAProxy documentation. The other canonical software balancer; its algorithms, stick tables, slowstart, and health-check model are the reference implementation of Sections 2–5 and 7.
• Google SRE Book: Load Balancing at the Frontend. DNS-based and VIP-based global balancing; behind Section 6.
• Google SRE Book: Load Balancing in the Datacenter. Subsetting and the client-side thinking behind Section 9.
• Amazon Builders' Library: Implementing health checks. The best treatment of the dependency-check trap in Section 4.
• Maglev: A Fast and Reliable Software Network Load Balancer (NSDI 2016). The stateless L4 design behind Section 6's hyperscale paragraph.
• Envoy: Outlier detection and Panic threshold. The ejection caps and fail-open rule from Sections 4 and 7, as shipped.
• Dean & Barroso: The Tail at Scale (CACM 2013). Hedged requests and why tails compound; behind Section 9.
• AWS: Elastic Load Balancing documentation. ALB (L7) vs NLB (L4) is Section 3 as a purchasing decision, with target-group health checks and deregistration delay as Section 4.
• Designing Data-Intensive Applications: Martin Kleppmann. The replication and partitioning chapters touch the consistent-hashing and request-routing ideas throughout.

Where you'll meet this

This post is the expanded version of the box the URL shortener interview drew in Step 3 and never explained. It was also hiding inside the resilience post (#2): outlier ejection (the balancer's version of a circuit breaker), retry budgets, and admission control are Sections 7 and 8 of this post seen from the application's side, and hedged requests started there. The sharding post's (#3) consistent hashing came back as the cache-affinity algorithm in Section 2. The CDN post's (#7) anycast is Section 6 at planetary scale. The rate-limiting post's (#8) Retry-After is what the balancer attaches to the 503 it sends when it sheds load in Section 8 (the 429 is the rate limiter's own verdict), and its per-IP limits are why Section 3 cares so much about X-Forwarded-For. The observability post (#9) is where the balancer's metrics and request IDs end up. Next up is security and abuse (#12), because the traffic cop also has to notice the pickpockets.

Keep exploring: the Core Concepts series

Every post in the series stands alone. Read them in any order.

• From Laptop to Billions of Clicks: Designing a URL Shortener in 11 Steps — the anchor: one design, every concept under load.
• #1 The 100:1 Superpower: Caching — the fastest request is the one you never make.
• #2 The Blast Radius: Surviving the Day Your Dependencies Fail — staying up when everything you depend on goes down.
• #3 Divide and Conquer: Sharding and Partitioning — splitting one database into many without losing your mind.
• #4 The Snowflake Problem: Unique IDs at Scale — naming things when millions are born every second.
• #5 Copies of the Truth: Replication — keeping copies of your data that actually agree.
• #6 Do No Harm Twice: Idempotency — making "just retry it" safe.
• #7 The Copy at the Doorstep: CDNs and Edge Computing — serving from next door instead of across the ocean.
• #8 The Bouncer's Math: Rate Limiting — saying no politely, at scale.
• #9 What Broke at 3 AM: Observability, p99, and Useful Alerts — knowing what's wrong before your users tell you.
• #10 Do It Later, On Purpose: Async Processing and Queues — the work the user doesn't have to wait for.
• #11 The Traffic Cop: Load Balancing — the box in every diagram nobody explains. (this post)
• #12 Assume They're Already Knocking: Security and Abuse at Scale — designing for the users who are designing against you.
• #13 Do the Math First: Estimation for System Design — the two minutes of arithmetic that choose the architecture.
• #14 The Whiteboard Playbook: Taking On Any System Design Challenge — the whole series in forty-five minutes.

Blueprints of Scale — Core Concepts #11. If this helped, the best thanks is a share with someone who's learning.

Here's what's covered: the launch-day story and why an API without limits is a tragedy of the commons; why limits exist (fairness, cost, survival) and the difference between a rate limit and a quota; the algorithms (token bucket, leaky bucket, fixed window, sliding window, and GCRA, the one underneath the token and leaky buckets) compared without flinching; where to enforce limits, which tools do it, and how to roll one out without breaking anyone; the hard part, one limit shared across many servers, and what to do when the shared counter is slow or gone; how to say no politely (429s, Retry-After, the headers, and what a good client does with them); the failure modes (key explosions, shared IPs, clock skew, stale limits, and how to watch the limiter itself); why rate limiting is not DDoS protection; and the principal-level toolkit: adaptive and concurrency limits, cost-based limiting, hierarchical limits, and shedding the right traffic first.

Sections 1 and 2 assume nothing. Sections 3 through 8 are the machinery. Section 9 is the judgment. The cheat sheet is at the end under The bouncer's math, distilled, and every diagram is also described in the text around it.

Section 1 — Launch day

Every rate-limiting story starts the same way.

A startup launches its public API on a Tuesday. By Wednesday morning, a price-comparison site has pointed a scraper at it. Polite by scraper standards, maybe, but it's asking for the full catalog, every product, every price, every fifteen minutes, from a hundred parallel connections. By Wednesday afternoon the API's response times have tripled. Real users, the ones with credit cards, are timing out at checkout. The on-call engineer finds the database CPU pegged, the connection pool (the fixed set of open database connections the servers share) exhausted, and one client responsible for more than 90% of all requests. The fix that morning is embarrassing in retrospect: there was no limit. Anyone could ask for anything, as fast as they could ask, forever. The scraper wasn't evil, just the first client to discover that the door had no bouncer.

The diagram is the genre in five boxes: healthy traffic, one unbounded client, a saturated database, and real users paying the price. (The p99 is the latency that 99% of requests come in under, the edge of the slow tail; the observability post, #9, is about why it's the number to watch.) The shape rarely varies. The failure is almost never "too many legitimate users," though flash sales and launches are real and Section 9's graceful degradation comes back to them. It's one client, a scraper, a runaway script, a retry storm from a buggy deploy, a partner who "just" put your endpoint in a tight loop, consuming the shared resource faster than everyone else combined. The API is a commons. Without limits, the commons gets grazed to dirt, and the tragedy isn't theoretical: it's your checkout page timing out while someone's price bot hammers your catalog for the fourth time that hour.

The naive fixes and why they fail: "add more servers" (the scraper scales faster than your budget, and you're now paying to serve someone else's business); "block the IP" (whack-a-mole; the next one uses a different IP, and meanwhile you've built a manual process for an automatic problem); "make the database faster" (the bottleneck moves, the scraper doesn't). The fix is not more capacity but deciding, per client, how much is enough, and enforcing it.

The bouncer doesn't hate the crowd; the bouncer is why the club stays open.

Section 2 — Why limits exist

Limits exist for three reasons, and they're worth separating, because each one argues for a different design:

1. Fairness: the noisy neighbor. Your API serves many clients from one pool of capacity. Without limits, one client's spike is every other client's outage. This is the multi-tenancy problem (a tenant is one customer sharing your infrastructure with others): your biggest customer and your smallest share the same database, and the smallest shouldn't go down because the biggest had a busy Tuesday. Fairness limits say nobody gets more than their share. The resilience post (#2) calls the structural version of this a bulkhead, a wall between tenants so one can't flood the other. A rate limit is a bulkhead with a number on it.

2. Cost: every request has a price. Compute, database reads, bandwidth, third-party API calls you pay per request. Traffic is money, and unlimited traffic is an unlimited bill. This one bites hardest with AI endpoints and anything that fans out to a paid dependency. A runaway client isn't just slow; it's expensive. Cost limits say nobody spends more of our money than we agreed.

3. Survival: the system has a ceiling. Every system has a maximum throughput past which it doesn't degrade gracefully. It falls over. Queues grow unbounded, latency explodes, retries pile on (the resilience post's retry storm), and recovery takes longer than the outage. Survival limits say we will shed load ourselves, on our terms, rather than let the load shed us. A 429 is a controlled "no." A timeout cascade is an uncontrolled one.

One shared pool, three things being protected: other tenants, your money, and the system's ability to stay upright.

The cost reason has a story that every team in this genre eventually tells about itself. A free tier with no limits, because "we'll add them when we need them." A GPU-backed image-resize endpoint, billed by the second. A botnet that discovers it will happily process ten thousand images a minute. The bill arrives before the monitoring alert does, and the postmortem contains the sentence: we knew the endpoint was expensive; we just never connected "expensive" to "unlimited." Limits go in that week: per-key daily caps on the expensive endpoints, generous per-IP limits on the cheap ones. The bots leave. The bill normalizes.

Notice that the fix in that story used two different kinds of limit, and they deserve different names. A rate limit smooths traffic over short windows: 100 requests per minute, 10 per second. A quota is a budget over a long window, tied to a plan: 5,000 API calls per hour (GitHub), a monthly allowance of image conversions, a daily cap on an expensive endpoint. They're stored differently (a rate limit's counter can live in memory and expire in seconds; a quota is durable and usually linked to billing), they fail differently (a rate limit says "slow down"; a quota says "upgrade or wait until the first of the month"), and they're enforced at different places. Most APIs need both. The rest of this post is mostly about rate limits, and Section 9 comes back to quotas.

The reframing that matters: a rate limit is a promise, not a punishment. "100 requests per minute" tells a developer exactly what they can rely on. They can build retries, backoff, and budgets around it. "As much as you want until we panic and block you" tells them nothing, and the blocking, when it comes, is arbitrary. Good limits are documented, consistent, and boring. Bad limits are a surprise IP ban with no explanation.

Most real systems run all three reasons at once: fairness per key, cost per expensive endpoint, survival at the edge. Section 4 is about where each one lives.

Section 3 — The algorithms, compared without flinching

Nearly every rate limiter ever built is one of a handful of algorithms. They all answer the same question, "has this client had enough?", but they keep score differently, and the differences matter.

Token bucket. Imagine a bucket that holds N tokens. Every request costs one token. Tokens drip back in at a steady rate, say 10 per second. Request arrives, token available? Take it, let the request through. No token? Reject. The bucket's depth is the point: a full bucket lets a client burst. 100 tokens in the bucket means 100 requests can go through right now, even though the refill rate is only 10 a second. That's the feature. Real traffic is bursty (a page load fires 30 requests at once), and the token bucket absorbs bursts while still capping the long-term average. The edge case: the bucket refills even when idle, so a client that goes quiet accumulates a full bucket and then spends it all at once. If that burst is a problem, shrink the bucket. The two knobs, rate and burst, are the pair you'll see in every limiter's configuration.

Leaky bucket. This one comes in two forms, and the names get mixed up. As a queue, requests pour into a bucket and the bucket leaks at a fixed rate, say 10 requests a second flow out to the server; arrivals faster than the leak fill the bucket, and when it's full the overflow is rejected. That version smooths traffic into a perfectly steady stream, which is what you want when the thing downstream can only handle a flat rate (a partner API with its own strict limit, a worker pool of fixed size), and its cost is that queued requests wait, so latency grows under burst. As a meter, the leaky bucket doesn't queue anything; it just tracks how full a hypothetical bucket would be and rejects when it would overflow, and in that form it makes exactly the same decisions as a token bucket, just phrased upside down. NGINX's limit_req is a leaky-bucket meter; add burst and requests queue up to the burst size, add nodelay and it behaves like a token bucket. So the real choice is "do I want to smooth (queue) or to cap (reject)?", not "leaky or token."

Fixed window. The simplest: divide time into windows, one minute each. Count requests per client per window. Hit 100 in the current minute? Reject until the next minute starts, when the counter resets to zero. Two numbers in memory per client. The edge case is famous: the window boundary. 100 requests at 12:00:59 and 100 more at 12:01:01 is 200 requests in two seconds, and the fixed window allows all of it, because they fell in different windows. If your system can't survive twice the limit for a few seconds, fixed window is the wrong tool.

Sliding window. Fixed window's flaw, fixed: instead of resetting at boundaries, look back over the last N seconds from now. "How many requests in the last 60 seconds?", counted continuously, no boundaries, no 2× spike. The cost is memory. A precise sliding window keeps a log of timestamps per client, and at high traffic that's a lot of timestamps: in Redis, each entry in a sorted set costs on the order of a hundred bytes once the skip-list and dictionary structures around it are counted, so a client doing 1,000 requests a second against a 60-second window holds around six megabytes of log, per key. The common compromise is the sliding window counter: keep the previous window's count and the current window's count, and estimate the rate as previous × (fraction of the previous window still inside the last 60 seconds) + current. Two numbers per client, no boundary spike, approximate. Cloudflare published the accuracy of this trick from their own traffic: across 400 million requests, 0.003% were decided wrongly and the average rate error was about 6%, with every misjudgment in the lenient direction: a few sources slipped through slightly above the threshold, and none was blocked below it. Most production "sliding window" limiters are this approximation.

GCRA, the one underneath. The generic cell rate algorithm is worth knowing because it's what several real limiters actually run (the redis-cell module's CL.THROTTLE command, for one). It keeps a single timestamp per client, the "theoretical arrival time" of the next allowed request, and each request either arrives after that time minus a burst tolerance (allowed; advance the timestamp by one interval) or too early (reject, and the difference tells you exactly how long to wait, which is your Retry-After for free). One number of state, no background refill process, and it's mathematically the token bucket and the leaky-bucket meter in one formula.

The comparison:

So which do you actually pick? Token bucket (or GCRA, which is the same guarantee) is the default; it matches how real clients behave: bursty, then quiet. Reach for the leaky-bucket queue when you're shaping traffic into something fragile that needs a flat rate. Fixed window is fine when the limit is coarse and a boundary spike is harmless: daily quotas, not per-second promises. And the sliding window log when the number is a contract, or the counter when "within a few percent, erring lenient" is close enough.

A story about picking wrong, so the table sticks, and it's a subtler story than it first looks. A login endpoint limited with a fixed window: five attempts per minute per IP. Credential-stuffing bots (replaying leaked username-password pairs; the security post, #12, has the whole attack) learn the boundary: five attempts at 12:00:59, five more at 12:01:01. Notice what that does and doesn't buy them. It doesn't raise their hourly throughput, which is 300 attempts per IP either way. What it does is let them fire ten attempts in two seconds once a minute, a burst the sliding window would have spread out. And the bigger problem sits one level up: the bots came from thousands of IPs, so a per-IP limit of any algorithm was the wrong key for this endpoint (Section 7). Switching to a sliding window counter halved their peak; switching the key to the target username is what actually protected the accounts. The algorithm is the guarantee, and the key is what the guarantee applies to. Pick both deliberately.

Section 4 — Where the bouncer stands

A rate limiter has to live somewhere. The three options, from outside in:

At the edge (CDN or load balancer). This is the coarsest, cheapest place: limits enforced before the request ever reaches your infrastructure. The edge says no to the obvious stuff, the scraper from Section 1, the botnet, the single IP doing 5,000 requests a second. It's cheap because the request dies early: no app server touched, no database queried. The edge's limitation is not what it can see (modern edge rules can key on headers, cookies, query parameters, a JWT's claims, the client's TLS fingerprint, or its network) but what it knows: it has no idea that this API key is on the enterprise plan unless you teach it, and teaching it means syncing your plan data to a third party. So edge limits tend to be survival limits (Section 2's third reason). They keep the building standing.

At the API gateway. One hop in, and now the request has identity: an API key, a user, a plan tier. The gateway is where fairness and cost limits live: "free tier: 100 a minute, pro: 10,000 a minute," "image-resize: 50 a day per key" (a quota in Section 2's terms, but enforced at the same door). The gateway is shared infrastructure, so the limit logic is written once and applies to every service behind it. This is where most of your rate limiting should live, because it's the one place that sees every client and every service.

Per service. The innermost layer, and the most precise: the service knows that POST /resize costs 100 times more than GET /status, so it limits them differently. Per-service limits protect specific expensive operations and encode domain knowledge the gateway doesn't have. The cost: every service implements (or imports) its own limiter, and the logic can drift.

Each layer protects against the failure of the others. The gateway's Redis goes down (Section 5)? The edge limits still blunt the flood. A new service ships without its own limiter? The gateway's per-key limits still apply. An attacker rotates IPs to dodge the edge? The gateway's per-key limits catch the key. Skipping a layer isn't simplicity; it's a single point of failure for "too much traffic."

The diagram has two boxes the usual version leaves out, because limits point outward as well as inward. A service that calls a third-party API has to respect their limit, so it runs a client-side limiter on its outbound calls (a token bucket sized to the partner's contract) rather than discovering the partner's 429s in production. And the connection pool in front of a shared database is a limit too, a concurrency limit rather than a rate limit, and often the one that actually decides who suffers when the database slows. Section 9 has more on those.

The tools, by layer, since "a rate limiter" is usually a configuration rather than code: at the edge, Cloudflare's rate-limiting rules, AWS WAF rate-based rules, and NGINX's limit_req and limit_conn modules; at the gateway, AWS API Gateway usage plans, Kong and APISIX plugins, and Envoy's global rate limit service (a small gRPC service backed by Redis that any Envoy proxy can ask "is this descriptor over its limit?"); in the service, a library (Bucket4j, golang.org/x/time/rate, limits in Python) or the Redis pattern in Section 5.

The commonest version of this story is beautiful per-service limiters, every endpoint tuned, every cost modeled, and then a new service launched without one. It takes one enthusiastic integration partner three days to discover the unprotected endpoint, and the resulting flood takes down the shared database, which degrades every other service, limiters and all. The gateway-level limit added afterward isn't as precise, but it holds the line.

Two rollout habits that keep a new limit from becoming its own incident. First, shadow mode: deploy the limit counting and logging but not rejecting, watch the "would have denied" numbers for a week, and find the legitimate integration you'd have broken before you break it (Envoy's rate limit filter has a knob for it, a runtime fraction that enforces versus merely observes; Stripe describes the same dark-launch practice). Second, overrides as a product surface: limits live in configuration with per-customer exceptions, and there's a documented way to ask for more. Stripe asks for six weeks' notice for limit increases; the point is that "we need a higher limit" is a ticket, not a heroic config change during the customer's launch.

Edge limits go everywhere; they're nearly free. Gateway limits are the workhorse: everything with an API key gets one. Per-service limits go on the expensive 1%, the endpoints whose cost is wildly different from the average. Per-service limits are precision instruments. Gateway limits are the seatbelt. Wear the seatbelt.

Section 5 — One limit, many servers

Everything so far assumed one bouncer with one clipboard. Real APIs run on many servers behind a load balancer, and each server sees only its own fraction of the traffic. Twenty servers, each allowing 100 requests per minute, is a 2,000-per-minute limit that calls itself 100. A local limiter on N servers is N times the limit. So the count has to live somewhere shared.

The standard answer is Redis: fast, in-memory, with operations that fit counting well. The pattern for a fixed window: INCR a key like ratelimit:{client}:{window}, set it to expire when the window ends, and reject when the count passes the limit. For a token bucket: store the token count and the last-refill timestamp together, and refill on read. A single Redis node handles well over 100,000 of these operations a second.

The section's real lesson is about atomicity, and it's worth being precise about where the races actually are, because the usual telling gets it slightly wrong. INCR by itself is atomic and returns the new value, so a fixed-window counter never needs a separate read: increment, look at the number that comes back, decide. The race in the fixed-window pattern is between INCR and EXPIRE: if the server dies between the two, the key never expires and that client is limited forever. The fix is to do both in one step, with MULTI/EXEC or a four-line Lua script. The other race, the one people mean when they say "check then act," is real for anything that needs a read-modify-write: a token bucket has to read the current tokens and the last refill time, compute the refill, decide, and write the new state, and two servers doing that in application code will both read the same state and both write. That one needs a Lua script (or Redis Functions in 7.0+), because Redis executes scripts atomically: server A's script runs completely, then server B's, and one of them sees the empty bucket. The script is the arbiter; the race is decided by Redis, not by your code. If this paragraph felt familiar, it's Section 5 of the idempotency post (#6) with the nouns changed. Distributed systems have a small number of races, and they show up everywhere.

The gap between "works" and "correct under concurrency" is one atomic step. A team that builds its token bucket as GET then SET in application code will pass every test, because tests don't race, and then find on the busiest day of the year that the limit leaks by roughly the number of servers that happened to read the same state at the same instant. With a shared counter that's a leak of a few percent, ugly but bounded. The 20× version of the story comes from the other mistake, running a local counter on each server and calling it a shared limit.

Three refinements worth knowing. Sliding-window log in Redis: store each request's timestamp in a sorted set, and on each request drop the timestamps older than the window and count what's left. Precise, but the set grows with traffic (Section 3's megabyte per busy key), so cap it and expire the key. Approximate counters: at very high scale, exactness is negotiable, and a structure like a Count-Min Sketch estimates "how many" in fixed memory. Its error is one-sided, always overestimating, so it may reject a client at 98 instead of 100 but never lets 102 through; for a survival limit that's the right direction to be wrong. Exact counting is a luxury; bounded memory is a requirement. Take the time from Redis, not the app server. Section 7 will explain why app-server clocks disagree; inside a Lua script you can call TIME and have Redis's clock be the one clock every server uses.

Now the operational reality, because the limiter is on the critical path: every request waits on a Redis round trip before proceeding. Four things follow.

• Budget the round trip. The call needs a tight timeout, a millisecond or two, and when it times out you need a decision (below), not a hung request.
• Hot keys are real. All of one tenant's traffic hashes to one key on one Redis shard, and a large tenant can saturate that shard while the others idle. Sharding the limiter by key across a Redis cluster spreads ordinary keys; a single enormous key needs its own treatment (split it into a few sub-keys and sum, or move that tenant to local buckets).
• Local buckets with periodic sync. For very high rates, each server keeps a local bucket and periodically borrows a batch of tokens from the shared counter ("lease me 1,000 tokens for the next second"), so the shared store sees one call per server per second instead of one per request. The limit becomes approximately shared and exactly fast. And for pure survival limits (per-server admission control), local-only is fine, because the thing being protected is the server itself.
• Fail open or fail closed. If Redis is down, you choose: allow everything (the limit disappears during the outage) or reject everything (the limiter becomes the outage). For fairness and cost limits, many APIs fail open; Stripe's limiter is documented to, and Envoy's rate limit filter defaults to it. An unenforced limit beats a self-inflicted total outage. For abuse limits (login attempts, password resets; Section 8), the calculus flips, and you fail closed or fall back to a local bucket, because the attacker is exactly who benefits from an open door. Make that choice in a design review, not during the incident, and wrap the limiter in a circuit breaker (the resilience post's tool) so a slow Redis degrades to the fallback instead of dragging every request with it.

Redis plus Lua is the default. Sliding-window logs when precision matters and traffic is moderate; approximate counters when the alternative is running out of memory; local buckets with leases when the shared store can't keep up.

Section 6 — Saying no politely

Being rejected is fine. Being rejected without explanation is what makes developers hate your API. The HTTP standard gives you the vocabulary for a polite no, and it costs almost nothing to use it.

429 Too Many Requests. That's the status code, defined in RFC 6585. Not 403, which says "forbidden," a permissions problem (GitHub does return 403 for some limits, which is the exception that annoys everyone). And not 503, which is its cousin and deserves a precise distinction: 429 means you exceeded your limit; 503 means we exceeded ours. A client that gets a 429 should slow itself down. A client that gets a 503 should back off because the server is overloaded regardless of who's asking, which is the load-shedding case in Section 9 and the load balancer post (#11). NGINX's limit_req rejects with 503 by default, which is a setting worth changing. One more rule from the RFC: a 429 must not be cached, so check that your CDN isn't storing one client's rejection and handing it to the next.

Retry-After. A header that says how long to wait before trying again, either as a number of seconds (Retry-After: 60) or as an HTTP date; clients have to parse both. Without it, the client guesses, usually wrong, usually too aggressively, which is how a rate-limited client becomes a retry storm. The RFC says a 429 may include it; my advice is stronger. Always send it. It's the difference between "go away" and "come back in a minute."

The rate-limit headers. Sent on every response, not just rejections, so the client always knows where it stands. There's a long-standing de facto family and a standard on the way, and they don't match:

• The de facto headers, with vendor variations: GitHub sends x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-used, and x-ratelimit-reset (an epoch timestamp), plus x-ratelimit-resource to say which limit applied. X (Twitter) uses x-rate-limit-* with a hyphen. Shopify's REST Admin API (now a legacy API; its GraphQL successor reports the budget in the response body) sends a single X-Shopify-Shop-Api-Call-Limit: 32/40. Envoy's headers use delta seconds for the reset.
• The IETF's httpapi working group has been standardizing this for years, and as of its 2026 drafts the fields are RateLimit-Policy (the quota and window, e.g. "default";q=100;w=60) and RateLimit (the remaining count and seconds to reset, e.g. "default";r=50;t=30). Earlier drafts used RateLimit-Limit, -Remaining, and -Reset, and that older shape is what most blog posts and libraries still show. It's still a draft, not an RFC; if you adopt it, track the version.

Whichever family you pick, the point is the same: the limit stops being a surprise and becomes a dashboard. A client can see "3 remaining" and choose to defer its low-priority sync rather than burn its last requests and eat a 429.

The diagram is the contract: the client always knows where it stands, the rejection comes with instructions, and the retry lands after the wait instead of inside a storm. What happens without the contract is predictable. A platform ships rate limits with bare 429s, no headers, no Retry-After, no documentation. Its biggest integration partner responds the way anyone would: retry immediately, in a tight loop, from every worker. The partner's retry storm does more damage than the original traffic spike. The fix isn't a better algorithm; it's three headers and a documentation page, and partner traffic smooths out within a day. The cheapest rate-limiting upgrade is communication. Most 429 pain isn't the limit; it's the silence around it.

GitHub and Shopify are the models for "headers on every response." Stripe is the model for something different: limits set high enough that well-behaved integrations never notice them (100 requests a second on live mode, 25 per endpoint, plus concurrency caps), and a 429 that carries a Stripe-Rate-Limited-Reason header saying which limit tripped, so the developer knows whether to slow down globally, on one endpoint, or reduce parallelism. Stripe doesn't send the remaining-count headers at all. Both approaches work; what they share is that the developer is never guessing. Document the limits on the same page as your authentication docs, because that's where developers actually look. A limit nobody can see is indistinguishable from an outage.

What a good client does with all this. The server's half of the contract is the headers; the client's half is how it retries, and you'll write clients as often as servers. Honor Retry-After exactly. When there isn't one, back off exponentially with full jitter: wait a random amount between zero and min(cap, base × 2^attempt), which AWS's analysis showed cuts total retry work by more than half compared to backoff without jitter, because it stops every client from retrying in the same instant. Give retries a budget (no more than three attempts per request, and retries no more than 10% of your total traffic, Google's rule) so a struggling server isn't buried under retries of its own failures. And run a token bucket on the client side sized to the server's documented limit, so you never send the request you know will be rejected. The resilience post (#2) has the full retry chapter; this is the rate-limiting corner of it.

Three client behaviors the server has to plan for: clients that ignore Retry-After and hammer anyway (escalate: longer windows, then a temporary block); clients that supply their own "priority" header and expect to be believed (never trust it; priority is something you assign, Section 9); and the subtle one, limits that differ for existing and nonexistent accounts, which lets an attacker enumerate your users by timing 429s. The security post (#12) has that one.

Section 7 — How limiters break

The bouncer can become the problem. Four ways, plus the question of how you'd know.

Key cardinality explosion. Every limiter stores state per key: per IP, per API key, per user. An attacker (or a bug) can generate unlimited distinct keys, a fresh IP per request from a proxy pool, a random API key per request. Each key gets its own fresh bucket, its own full limit. Your per-key limit is now meaningless, and worse, the key space grows unbounded; memory fills with buckets for keys that will never be seen again. Defenses: bound the key space (limit by something the client can't mint freely, which means authenticated identity rather than IP wherever authentication exists), put a TTL on every key so idle ones expire, and cap the limiter's memory with an eviction policy (in Redis, set maxmemory and maxmemory-policy allkeys-lru, and watch the eviction counter). A limiter that grows without bound is a memory leak with a config file.

NAT and shared IPs: punishing the innocent. A corporate office, a university, a mobile carrier's carrier-grade NAT (CGNAT, where a whole region's phones share a few public addresses): hundreds or thousands of real users behind one IP. Per-IP limiting sees one "client" doing the traffic of a thousand. One user's scraper script gets the whole office 429'd. Per-IP limits are guilt by association. And IPv6 has the opposite problem: a single host can mint 2⁶⁴ addresses in its own /64 block, so limiting per IPv6 address is limiting per nothing; limit per /64 at minimum.

Two normal users, one runaway script, one shared IP, and the limiter punishes all three equally. That's the whole argument for choosing the key deliberately:

• Per tenant or per API key when keys map to integrations: one partner, one budget, revocable independently. This is the fair unit for a business API.
• Per user when the tenant is a person, inside a product.
• Per target for abuse endpoints: login attempts limited per username being attacked, not per attacker IP, so a distributed attack on one account is still caught (Section 3's story).
• Composite keys for unauthenticated endpoints, where there's no identity to key on: IP plus TLS fingerprint plus user agent, which is harder to mint than any one of them.
• Per IP only as the coarse outer layer, set high enough that only real abuse trips it. The IP limit is the fire code, not the seating chart.

Clock skew. Fixed and sliding windows depend on time: window boundaries, reset timestamps, token refill math. Servers whose clocks disagree by seconds compute different windows for the same client; one server's "new window" is another's "still the old one." The limit becomes inconsistent, and debugging it is miserable because each server's logs look correct locally. NTP everywhere is table stakes (NTP is the protocol that keeps machine clocks synchronized), but it's not the fix, and neither is a monotonic clock, which counts from each machine's boot and means nothing across machines. The fix is to have one clock decide: read now from Redis inside the script (redis.call('TIME')), or lean on key TTLs, which Redis expires by its own clock. If the counter and the clock live in the same place, skew can't split them.

Limits that never get revisited. The quietest failure mode: the limit someone set in 2021 (100 a minute, seemed fine) is still there in 2026, and the product has tripled in complexity. Legitimate integrations start getting 429s on normal usage. Developers learn that the documented limit is a lie and build aggressive retry logic "just in case," which makes the traffic burstier, which makes the limit more necessary. A stale limit trains your clients to misbehave. Limit values live in configuration with a review date, not in code with a commit from three years ago.

The webhook version of the shared-IP story is worth telling because the key is on the other side. A SaaS product rate-limits its outbound webhook delivery per destination IP, which is sensible until its biggest customer puts 400 employees and all their internal tools behind one NAT address. Every deploy announcement (hundreds of webhooks in a minute) trips the per-IP limit, and the customer's team misses deploy notifications for weeks before anyone connects "we didn't get the webhook" to "your office shares an IP." The fix is per-customer keys instead of per-IP. The key should identify the tenant, not the network path.

How you'd know. A limiter with no dashboard is a limiter that fails silently in both directions. Graph, per limit: the allow and deny rates (and alert on a change in the deny rate, up or down, because a drop to zero means the limiter stopped working); the top denied keys, which is where you find the partner who's about to email you; the fraction of keys running above 80% of their limit, which is your early warning that a limit is stale; the limiter's own round-trip latency at the p99; and the count of fail-open events. In shadow mode (Section 4), the "would have denied" rate is the number that tells you whether the new limit is safe to enforce. The observability post (#9) has the alerting rules; these are the signals to feed them.

Key design gets revisited whenever you add authentication: migrate IP keys to identity keys. Clock discipline is forever. Limit values go on a schedule: quarterly for fast-moving products, annually at minimum. A limiter is infrastructure, and infrastructure rots.

Section 8 — Rate limiting is not DDoS protection

Rate limiting and DDoS protection get confused constantly, because both say "too much traffic, go away." They're different tools for different adversaries.

Rate limiting assumes a legitimate client going too fast. An API key, a real integration, a scraper with a business model. The client wants to use your API; it just wants more than its share. The response is polite and precise: 429, Retry-After, headers, documentation. The goal is shaping. Keep the client, lose the excess.

DDoS protection assumes an attacker trying to take you down. A distributed denial-of-service attack sends traffic whose only purpose is to make you fall over: SYN floods (opening millions of connections that never finish their handshake), reflection attacks (small forged requests to public servers that reply with big responses aimed at you), botnets of hijacked machines. There's no API key to limit, no "share" to be fair about. The response is blunt: drop the traffic at the network edge, challenge suspicious clients (a JavaScript challenge that a real browser passes and a script can't, then a CAPTCHA), absorb the flood across a global anycast network (the CDN post, #7, explains how announcing one address from 300 cities spreads an attack across all of them). The goal is survival. Keep the service up, lose the attacker.

The confusion costs real money in both directions. Relying on rate limiting for DDoS: your per-IP limiter sees ten thousand IPs each doing 10 requests a second, every one under the limit, collectively a flood of 100,000 a second. The limiter waves them all through while the site goes down. Relying on DDoS protection for rate limiting: your edge drops the flood but has no concept of "this API key gets 100 a minute," so the legitimate-but-greedy integration partner keeps hammering, and there's no polite 429 telling them to slow down, just an eventual IP ban that takes a paying customer offline.

The clean split holds for the volumetric attacks, the ones that work at the packet and connection level. For application-level floods (a botnet hitting your search endpoint with plausible-looking HTTP requests), the edge's first response is, in fact, a rate-limiting rule, keyed on whatever the edge can see: per IP, per network, per fingerprint. Cloudflare's own learning-center page lists DDoS, brute force, and scraping as uses of rate limiting. So the honest version is: rate limiting is one of the tools the edge uses against L7 attacks, and none of the tools it uses against L3/L4 floods, and neither of those is the fairness-and-cost limiting this post is mostly about. Three jobs; overlapping tools; keep the jobs distinct.

A company that tells its enterprise prospects "we have rate limiting, so we're protected against DDoS" is going to have a specific kind of bad afternoon. During a modest attack, a few gigabits, the per-IP limiter does exactly what it was designed to do: nothing, because no single IP exceeds the limit. Four hours later the postmortem's first line reads rate limiting is about fairness among customers; DDoS protection is about survival against attackers, and we had confused the two. They buy proper edge mitigation the next quarter, and the rate limiter stays, doing its actual job.

And one more job that lives near this one but belongs to the security post (#12): throttling that exists to stop abuse rather than overload. Login attempts, password resets, one-time-code requests, signups, "send me an SMS" endpoints (which attackers use to run up your SMS bill, a scheme called SMS pumping). Those limits are keyed per target account as well as per source, they slow the client down rather than just capping it, they fail closed, and they're part of a detection system, not just a fairness one. Same mechanism, different purpose, different defaults.

Rate limiting is for every API, always; it's about your customers. DDoS protection (Cloudflare, AWS Shield, an equivalent) is for when availability matters enough to pay for; it's about your enemies. Abuse throttling is about the users who aren't who they say they are. The edge absorbs the attack; the limiter shapes the customers; the abuse controls watch the door.

Section 9 — Going deep: the principal-level toolkit

Fixed limits have a flaw: they're set for a capacity guess. Traffic grows, the guess goes stale (Section 7), and someone has to notice. The principal-level move is closing the loop: limits that follow the system's real capacity instead of a number someone wrote down last year. And then a set of judgment calls about what "fair" means when there isn't enough to go around.

Concurrency limits, static and adaptive. A rate limit counts requests per unit of time. A concurrency limit counts requests in flight at once, and it's the better tool for protecting a resource, because a slow request costs the same as ten fast ones under a rate limit and ten times as much under a concurrency limit. The static version is a semaphore: at most 30 concurrent payouts per account (Stripe does exactly this), at most 50 connections in the database pool, at most N in-flight calls to a fragile partner. That's the bulkhead from Section 2 with a number on it, and it's often the limit that actually decides who suffers when the database slows. The adaptive version watches the system's health and moves the number. Netflix's concurrency-limits library is the reference design, and it ships several algorithms borrowed from TCP's congestion control: AIMD (add one to the limit while requests are healthy, multiply it by 0.9 when one times out or errors), and the latency-gradient family (Vegas, and Gradient2, which the library's README says smooths out outliers in bursty traffic), which shrinks the limit when the observed latency starts rising above its recent minimum. The limit follows the system's real capacity instead of a stale config value. When the database slows, the limit tightens by itself; when you add capacity, it loosens. No calendar reminder required.

Two relatives worth knowing by name. Google's client-side adaptive throttling (from the SRE book's overload chapter) has each client track its own requests and accepts, and start rejecting locally when requests exceed about twice the accepts, so a rejected-everywhere client stops sending before it's told to. And CoDel-style queue timeouts drop the requests that have already waited longer than the system could ever serve them usefully, which is the resilience post's (#2) territory.

Hierarchical limits. Real limits nest: a global ceiling, then per tenant, then per user within the tenant, then per endpoint. A request has to pass every level. Two rules keep this from producing mysteries. The children's limits should sum to no more than the parent's, or the parent limit is fiction. And charging should be all-or-nothing: if a request passes the tenant limit but fails the endpoint limit, the tenant's token has to be refunded, or clients get charged for requests that were rejected, which is a bug that's very hard to see from the outside. Layered limits are how you express "fair, but the costly stuff is fairer."

Quotas, revisited. Section 2 separated quotas from rate limits. The quota's implementation is different enough to say plainly: it's durable (a counter in the database, not in a Redis key with a TTL), it's tied to the billing plan, its window is long (a day, a month), its "reset" is a business rule, and its rejection is a product moment ("you've used your 10,000 conversions; upgrade?") rather than a transport one. GitHub's 5,000 requests per hour is a quota; Stripe's 500 reads per transaction over 30 days is a quota. Design the upgrade path before the rejection, because the rejection is where customers decide whether to pay you more or leave.

Cost-based limiting: not all requests cost the same. GET /status costs microseconds. POST /report?format=pdf&years=10 costs seconds of compute and a database scan. Counting them both as "one request" is a lie the limiter tells itself; ten thousand cheap requests and ten thousand expensive ones are different floods. The fix is to assign costs. Each endpoint declares its weight (status = 1, report = 500), and the bucket holds cost units, not requests. A client's burst of cheap calls barely dents the bucket; one expensive call takes a real bite. The clearest production examples are the GraphQL APIs, where the cost of a query can't be known from its URL: Shopify computes a calculated cost per query (100 points a second on the standard plan, 1,000 on Plus, 2,000 on Enterprise, with a 1,000-point ceiling per query) and refunds the difference between the estimate and the actual cost after the query runs; GitHub's GraphQL API uses a similar points model; Cloudflare's enterprise rate limiting can score request complexity. It's more bookkeeping, and it's the difference between a limit that protects and a limit that performs.

Fairness is not the same as caps. Every tenant can be under its own cap and the sum can still exceed what the system can serve, and at that moment "everyone under cap" gives you nothing. Real fairness under overload is a share: max-min fair allocation (satisfy the smallest demands fully, split what's left among the large ones) or weighted fair queueing, where each tenant gets a weighted slice of the actual capacity rather than a fixed number. Google's answer is per-customer quotas denominated in CPU-seconds rather than requests, enforced at overload. Most teams never need this. The ones who do are the ones with one customer big enough to be everyone else's outage.

Limit the misses harder than the hits. Most abuse produces failures, not successes. Someone guessing short links gets 404s, someone stuffing credentials gets 401s, someone probing an API gets validation errors, while a legitimate client's miss rate is close to zero. So count misses separately, per client, and give them a far tighter budget than the overall limit: a thousand requests a minute is fine, ten failed lookups a minute is not, and after that the client gets a 429 (or a slow, deliberately identical response, so that the presence or absence of a key can't be read from the timing). This miss-rate limit is the cheapest enumeration defense there is, it's the one the URL shortener reaches for in Step 11 and the security post (#12) leans on in its Section 2, and it composes with everything above: the same bucket algorithm, a different counter, keyed on the same identity.

Graceful degradation: shed the right traffic first. When the system is overloaded despite every limit, and one day it will be (Section 1's flash sale, the launch that outruns the forecast), something gets dropped. The principal-level question is what. Not all traffic is equal: the checkout request matters more than the recommendation refresh; the paying customer's sync matters more than the free tier's; writes you've already acknowledged matter more than speculative prefetches. Design the shed order in advance: priority tiers, with the shedder dropping tier 3 before tier 2 before tier 1. Google's SRE book propagates a criticality label with every request (CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE), carried in the RPC metadata (the headers that travel with each call) so every service down the chain sheds consistently; their balancers treat retries like any other request, which is exactly why the retry budgets from Section 6 exist. Amazon's Builders' Library makes the complementary point: measure goodput, the requests that complete usefully, not throughput, because under overload the two diverge and the second one lies. Chosen victims include nothing important. Random victims include everything.

The final reframe. Rate limiting is usually taught as a defensive trick, a bouncer keeping the riffraff out. It's bigger than that: it's how a system expresses its priorities under pressure. Every limit is a statement about what matters: this tenant matters this much, this endpoint costs this much, this traffic gets shed first. The bouncer isn't just counting heads. The bouncer is enforcing the fire code, the seating chart, and the VIP list, all at once, at ten thousand decisions a second. Design your limits like you'd design your values, because under load, they are.

The bouncer's math, distilled

For the skimmers and the revisitors: everything above, on one page.

Key numbers and rules:

Every trade-off, in one table:

Three ideas to take with you:

1. A rate limit is a promise, not a punishment. Documented, consistent, boring limits let developers build retries and budgets around them. "As much as you want until we panic" is not a limit; it's a surprise outage with extra steps.

2. Every distributed limit bottoms out in an atomic step, and in one clock. A Redis script, a conditional write, one indivisible operation with one source of time. If your read and your write are two steps on two servers with two clocks, you don't have a limit, you have a suggestion.

3. Under load, your limits are your values. What you shed first, who gets the bigger bucket, which endpoint costs more: those are priority decisions expressed as arithmetic. Make them deliberately, in a design review, before the traffic makes them for you.

Further reading

• Cloudflare: What is rate limiting?. The clearest plain-language overview; behind Sections 1–2.
• Cloudflare: How we built rate limiting capable of scaling to millions of domains. The sliding-window-counter formula and its measured accuracy; behind Section 3.
• Cloudflare Docs: Rate limiting rules. Production rule configuration at the edge, including what the edge can key on; the Section 4 edge layer in practice.
• Stripe: Rate limits. Generous limits, concurrency caps, and the Stripe-Rate-Limited-Reason header; behind Sections 6 and 9.
• Stripe: Scaling your API with rate limiters. The token-bucket-in-Redis implementation, the fail-open decision, and the dark-launch rollout; behind Sections 4 and 5.
• GitHub: Rate limits for the REST API. The headers-on-every-response model, and a quota (5,000/hour) in the wild.
• IETF httpapi: RateLimit header fields for HTTP (draft). The RateLimit / RateLimit-Policy fields in Section 6; check the current draft number.
• redis-cell. GCRA as a Redis module, one atomic command; Section 3's fifth algorithm in production.
• NGINX: Limiting access with limit_req. The leaky-bucket meter (and its burst/nodelay knobs) on the web server behind close to a third of the web's sites; Section 3 in production clothing.
• Netflix: concurrency-limits. The adaptive concurrency library and its AIMD, Vegas, and Gradient algorithms; behind Section 9.
• Google SRE Book: Handling Overload. Per-customer quotas, criticality levels, and client-side adaptive throttling; behind Section 9.
• Amazon Builders' Library: Using load shedding to avoid overload. Goodput versus throughput, and shedding deliberately; behind Section 9.
• AWS Architecture Blog: Exponential Backoff and Jitter. The analysis behind "full jitter" in Section 6's client advice.

Where you'll meet this

This post is the resilience post's (#2) bulkhead with a number on it: Section 2's fairness limit is the bulkhead pattern enforced per client instead of per dependency, and the client-side retry rules in Section 6 are that post's retry chapter seen from the server. It leans on the idempotency post (#6) twice: the atomic Redis script in Section 5 is that post's check-then-act race, and a 429 with Retry-After only works if the retried request is safe to repeat, which means rate-limited endpoints should be idempotent endpoints. The edge layer in Section 4 is the CDN post's (#7) and the load balancer's (#11) territory, and the 503-versus-429 distinction is where this post hands off to load shedding at the balancer. Section 8's abuse throttling belongs to the security post (#12). And in the URL shortener design: what stops someone from minting a billion short URLs? A per-key limit on the create endpoint (cost-based; writes are the expensive ones), a generous per-IP limit on redirects (the cheap ones), and the edge survival limit in front of everything. That's the whole post, applied to one design. Next up is observability (#9), because a limiter you can't see is a limiter you can't trust.

Keep exploring: the Core Concepts series

Every post in the series stands alone. Read them in any order.

• From Laptop to Billions of Clicks: Designing a URL Shortener in 11 Steps — the anchor: one design, every concept under load.
• #1 The 100:1 Superpower: Caching — the fastest request is the one you never make.
• #2 The Blast Radius: Surviving the Day Your Dependencies Fail — staying up when everything you depend on goes down.
• #3 Divide and Conquer: Sharding and Partitioning — splitting one database into many without losing your mind.
• #4 The Snowflake Problem: Unique IDs at Scale — naming things when millions are born every second.
• #5 Copies of the Truth: Replication — keeping copies of your data that actually agree.
• #6 Do No Harm Twice: Idempotency — making "just retry it" safe.
• #7 The Copy at the Doorstep: CDNs and Edge Computing — serving from next door instead of across the ocean.
• #8 The Bouncer's Math: Rate Limiting — saying no politely, at scale. (this post)
• #9 What Broke at 3 AM: Observability, p99, and Useful Alerts — knowing what's wrong before your users tell you.
• #10 Do It Later, On Purpose: Async Processing and Queues — the work the user doesn't have to wait for.
• #11 The Traffic Cop: Load Balancing — the box in every diagram nobody explains.
• #12 Assume They're Already Knocking: Security and Abuse at Scale — designing for the users who are designing against you.
• #13 Do the Math First: Estimation for System Design — the two minutes of arithmetic that choose the architecture.
• #14 The Whiteboard Playbook: Taking On Any System Design Challenge — the whole series in forty-five minutes.

Blueprints of Scale — Core Concepts #8. If this helped, the best thanks is a share with someone who's learning.

This post is that two minutes, expanded to full size. Back-of-the-envelope estimation is the discipline of turning vague product statements ("a lot of users") into concrete numbers (requests per second, terabytes, megabits) before you design anything: close enough to choose the right architecture, rough enough to do on a napkin. Every senior engineer you've watched take apart a design interview does this first. It isn't a party trick; it's how you avoid building the wrong system confidently.

What's here: why estimation decides the architecture before any box is drawn; the unit toolkit (powers of ten, seconds per day, bits versus bytes) and the cheat sheet of sizes every engineer should carry; the latency numbers, with the humanized scale that makes them stick and the rows that have aged; traffic math, including peak factors, fan-out, and the difference between users and requests; storage math from record size through replication, indexes, compression, and growth; bandwidth, egress, and the cross-zone line item everyone forgets; memory and the "does it fit in RAM?" question, asked properly; latency budgets and what percentiles actually do when you add them; Little's law and the queueing curve behind the 60% rule; the traps that catch the careless; and the principal-level judgment: cost per million requests, capacity with headroom, load testing, growth modeling, and knowing when to stop calculating.

If you're new to this, Sections 1 and 2 need no background at all. Sections 3 through 8 are the working math. Section 9 is the judgment. There's a cheat sheet at the end under Do the math first, distilled, and every diagram is also described in the text around it.

Section 1 — The number that bent the design

Go back to the interview's Step 2, because it's the cleanest demonstration of the whole discipline. The requirement was "100 million new URLs a month." That sentence means nothing to an architecture; it's a business number. The estimation turned it into engineering numbers:

• 100,000,000 URLs ÷ 30 days ÷ 86,400 seconds ≈ 40 writes per second. (38.6, if you carry the digits, which you shouldn't.) A single database does that without noticing.
• The read-to-write ratio was 100:1. So 40 × 100 = about 4,000 reads per second. That's the number that matters.

Now, the reasoning that goes with that number, because it's easy to state it wrong. Four thousand primary-key lookups a second is not beyond one database. A Postgres with its hot rows in memory will serve tens of thousands of indexed reads a second on modest hardware. What the number tells you is different and more useful. First, the full dataset (Section 5 will show it's about 3 TB) won't fit in the database's memory, so a fraction of those reads will hit disk and drag the tail latency with them, while the hot 20 million keys fit comfortably in a 10 GB cache where every read is a sub-millisecond hit. Second, the database's headroom is precious: it's what absorbs the writes, the replication, the failover, and the reporting query someone runs at noon, and you don't want 4,000 reads a second spending it. Third, a cache read costs a small fraction of a database read in dollars and in microseconds. So the design bends toward a cache, but the reason is cost, latency, and headroom, not inability. That distinction is what an interviewer is listening for. Say "the database can't handle it" and you've said something false; say "the database could, but this is cheaper and faster and keeps the primary's headroom for the writes" and you've said something true that took two minutes to derive.

That 4,000 also sized everything downstream. The key generation service (the KGS from Step 6, which hands out blocks of unused short keys) needs blocks of what size? At 40 writes a second, a block of 10,000 keys lasts a single server about four minutes; make it a million and refills become rare enough to ignore. The database: a single primary with a replica or two (the replication post's world, #5) holds for years, and sharding (#3) waits for the day writes grow a hundredfold. One estimate, three architecture decisions, zero meetings.

Notice what the estimation did not do: it didn't give a precise answer. Forty writes a second might really be 37 or 52. Nobody cared. The estimate's job is to sort possibilities into buckets (trivial, manageable, scary, impossible) and 4,000 reads a second landed in "manageable without a cache, comfortable with one." That bucketing is the whole game. You aren't computing the answer; you're ruling out the wrong architectures.

The costly version of skipping this step is a design that's correct and wrong at the same time. Picture a social product where every post is written into every follower's feed at post time (fan-out on write: do the expensive distribution once, when the post is created, so reads are cheap). "Millions of users," someone said, and the design went in. Then someone does the math after launch: a handful of accounts have followers in the millions, so a single post from one of them generates millions of writes in seconds, and the write path melts on the first viral day. The fix, fan-out on read for the big accounts (do the distribution when the feed is opened instead), is what two minutes of arithmetic would have prescribed on day one. The architecture was wrong in a way that multiplication would have caught.

So the discipline, stated once: before the first box, convert the business numbers into per-second, per-byte numbers, and let those numbers choose the shape. Everything in this post is the toolkit for doing that in minutes, on anything from a napkin to a whiteboard.

Section 2 — Five numbers and a napkin

Estimation runs on a tiny toolkit. Memorize it and you can do most system math in your head.

Powers of ten. A thousand, a million, a billion, a trillion: 10³, 10⁶, 10⁹, 10¹². In bytes: KB, MB, GB, TB, then PB at 10¹⁵. Pedants will note that a kilobyte is 1,024 bytes; estimation doesn't care. On a napkin, 1,000 = 1,024. The only place the 2.4% per step matters is that it compounds: by the time you reach terabytes it's a 10% difference, which is still inside the error bars of everything else you're doing.

Seconds per day: 86,400. Memorize this one number. Round it to 10⁵ for mental math (that's 8.6 × 10⁴ rounded up, so it understates rates by about 14%; fine for buckets). A month is about 2.6 × 10⁶ seconds. These two conversions turn every business metric into per-second math:

• 10 million requests a day ≈ 10⁷ / 10⁵ = about 100 requests per second.
• 100 million URLs a month ≈ 10⁸ / 2.6 × 10⁶ ≈ about 40 a second. There's the interview's number.

Bits versus bytes. Networks are sold in bits per second; storage and memory are sold in bytes. One byte is eight bits, so a "1 Gbps" link moves 125 MB/s, not 1 GB/s. This single confusion has sunk more bandwidth estimates than any other mistake in the field. When a number crosses the network/storage boundary, divide or multiply by 8, and say the units every time.

Round aggressively. Estimation uses one significant figure: 4,137 becomes 4,000, 86,400 becomes 10⁵. The point isn't sloppiness; it's matching the precision of your inputs. Your inputs ("about 100 million users") have one significant figure of truth in them, and carrying six digits through the math is theater. Round at the start, not at the end. (This post keeps a second digit in a few places, like "262 ms," where the point is to show the arithmetic. When you're deciding, one digit.)

The sizes you're expected to know. Interviewers and design reviews assume a shared set of magnitudes, and they're worth having on a card until they're in your head:

That's the toolkit: powers of ten, seconds per day, bits versus bytes, the discipline to round, and a card of sizes. Everything from here is applying it.

Section 3 — The cheat sheet in every senior's head

Every senior engineer carries roughly the same table in their head: the time it takes a computer to do basic things, from a CPU cache reference to a packet crossing an ocean. Peter Norvig published the first version around 2001, Jeff Dean's talks at Google made it famous, and Jonas Bonér's gist is the version everyone links. Here it is, with the humanized column that makes it memorable, where one nanosecond is stretched to one second:

Read the ratios, not the absolutes. Memory is 200 times slower than L1 cache. A disk seek is 100,000 times slower than a memory read. A round trip across an ocean is three hundred million times slower than an L1 hit. Every performance story in computing is a story about these ratios. The caching post's (#1) entire premise ("keep a copy close") is the observation that memory is hundreds of times faster than anything after it. The replication post's async-versus-sync agony is the 150 ms row: cross-continent consensus costs a bachelor's degree per round trip, so you only pay it when correctness demands it.

The humanized column is the part that sticks. Tell a junior "150 milliseconds" and they nod. Tell them "if an L1 hit is one heartbeat, that packet to the Netherlands takes a bachelor's degree" and they never forget it.

Two rows have aged, and it's worth knowing which. The disk-seek row is a museum piece; spinning disks are gone from most serving systems. And the SSD row is from the SATA era: a modern NVMe drive does a random 4 KB read in 10 to 100 microseconds, so "a normal weekend" is now more like "a long night's sleep." The ratios that matter (memory ≫ SSD ≫ network ≫ cross-region) have held for twenty years and will hold for twenty more. Memorize the ratios; re-check the absolutes every few years.

Since the table is old, here are the rows a working engineer in 2026 adds to it:

Two of these get used daily, so pull them out: a same-data-center round trip is a few hundred microseconds, call it half a millisecond (the unit of "talking to another server nearby"), and a cross-region round trip is 30 to 150 milliseconds depending on the geography (the unit of "talking to another continent"). Every latency budget in Section 7 is built from these.

Here's how the table gets used in practice. Someone asks why the p99 is 800 ms (the p99 is the time within which 99% of requests finish, so it's the experience of your slowest one-in-a-hundred; Section 7 has more). You walk the request path with the table in hand: three sequential database queries at 5 ms each is 15 ms; a cache miss adds a few more; and then there's a cross-region call to the payments provider at 150 ms. That's 170 ms, not 800, so the table has just told you something: the arithmetic doesn't reach 800 unless that cross-region call happens more than once, or times out and retries. You go look, and the payments client is configured with a 300 ms timeout and two retries, so the slow 1% of requests pay 300 + 300 + 150, roughly 750 plus the queries. There's your 800. The table doesn't tell you the answer. It tells you which explanations are arithmetically possible, which is most of the way there.

The fastest request is the one you don't make. The second fastest is the one answered from memory.

Section 4 — How many per second?

Traffic estimation is one question asked three ways: how many requests per second on average, how many at peak, and how are they split between reads and writes? (People write it as rps or QPS, requests or queries per second; same thing.)

From monthly totals, divide by 2.6 million seconds. The interview's 100M URLs a month → 40 a second. A billion events a month → about 400 a second. From daily totals, divide by 100,000. 10 million requests a day → about 100 a second.

From daily active users, multiply DAU by actions per user per day and divide by 100,000. A social app with 100M DAU where each user makes 50 requests (opens, scrolls, likes) → 5 billion requests a day → about 50,000 to 60,000 requests a second on average. That's a real system's number, computed in ten seconds.

Average is not the load you provision for; peak is. Traffic has a daily curve, and the peak hour runs two to three times the average for most consumer systems. The multiplier depends on the shape: business software spikes four or five times during working hours and idles at night; social networks peak two to three times in the evening; launches, sales, and live events hit ten times without apology; a notification service that releases ten million digests at 9:00 sees a peak that has nothing to do with its daily average at all. Pick the factor from the system's shape, not from optimism. So the pipeline is: compute the average, multiply by the peak factor, and that is the number capacity planning uses. 50,000 rps average → plan for 150,000. Section 9 turns this into hardware.

Then count the requests at every layer, not just the edge. A single page load that fans out to twenty backend services turns 50,000 rps at the edge into a million rps inside the system, and during an incident, retries multiply it again (the resilience post, #2, has that arithmetic). If you only count what arrives at the front door, your internal services are sized by accident. A related conversion people get wrong: "users online right now" is not requests per second. If 100,000 people are on the site and each one does something every 30 seconds on average, that's 100,000 ÷ 30 ≈ 3,300 rps. Requests per second equals concurrent users divided by the think time between actions.

Then split reads from writes. The ratio decides the architecture; the interview's 100:1 made it a caching problem. Social feeds are around 100:1. A payment system might be 10:1. And the ratio inverts. A logging pipeline takes something like a hundred writes for every read: a million events a second in, a handful of queries out. The interview's instinct ("cache the reads") is useless here, because there's nothing worth caching. The estimation instead sizes the write path: 1M events/s × 1 KB = 1 GB/s of ingest, which is the async processing post's world (#10) of buffers, batches, and backpressure. Read-heavy: cache. Write-heavy: buffer and batch. Balanced: honest database work in between. One caution that the ratio alone doesn't carry: 100:1 at 10 requests a second needs no cache at all. The ratio tells you which path to optimize; the absolute volume tells you whether you need to.

Always ask for the ratio before drawing anything. If nobody knows it, guess 10:1 and say the guess out loud, so it can be challenged.

The common failure: someone estimates "a million users" and provisions for a million simultaneous requests. A million users at 50 requests a day is 500 to 600 requests a second. Users are not requests: convert first, provision second.

Section 5 — How much disk?

Storage math is one multiplication with five factors. Miss any of them and the answer is fiction:

record size × number of records × growth window × replication factor × overhead

Work it with the URL shortener, since the numbers are already on the table:

1. Record size: 500 bytes per mapping (short key, long URL, two timestamps, an owner ID). Estimate record sizes from the schema, not from guesses: write down the fields and the card from Section 2.
2. Number of records: 100M new URLs a month.
3. Growth window: 5 years of retention → 100M × 60 months = 6 billion records.
4. Replication factor: 3 copies (the replication post's standard), because the raw number is never the stored number.
5. Overhead: indexes, metadata, filesystem slack. Add 30–50%. A database with a couple of indexes routinely stores 1.3 to 1.5 times the raw data, and each index is roughly rows × (key size + pointer) on top.

The math: 500 bytes × 6 × 10⁹ = 3 × 10¹² bytes = 3 TB raw. × 3 replicas = 9 TB. × 1.3 overhead ≈ about 12 TB. "Low terabytes," which is what the interview concluded, now with the working shown.

The two factors people forget are always the same: the replication factor and the overhead. The shape this takes is a new event store sized at the raw number, 40 TB, a nice round figure, budget approved. Then replication (×3) and index overhead (×1.4) turn 40 TB into 170, and the storage purchase covers a quarter of reality. The postmortem line writes itself: we estimated the data, not the database. Estimate the database.

There's a mirror-image mistake at the other end, and Section 9 will walk into it deliberately: object storage (S3 and its cousins) already stores your bytes redundantly across several data centers and bills you for the logical bytes you uploaded. Applying ×3 replication to an S3 estimate triples a bill that doesn't exist. Know which kind of storage you're estimating.

Three more multipliers that hide in the overhead factor and deserve their own line when they're large:

• Compression. Logs, JSON, and event streams compress five to ten times with zstd; columnar formats like Parquet compress analytics data five to twenty times. A petabyte of raw events is often 100 to 200 TB on disk. Apply it after the raw math, and say you did.
• Write amplification. Log-structured storage engines (the LSM trees under Cassandra, RocksDB, and most modern key-value stores) rewrite data during compaction, so a byte written by the application can become ten to thirty bytes written to disk over its lifetime. That's a disk-throughput factor, not a capacity one, and it's why write-heavy systems size their disks by IOPS as much as by gigabytes.
• Backups. Snapshots and archives are another one to two copies, on cheaper storage, but they're a line item.

And disk isn't only capacity; it's also IOPS (I/O operations per second) versus throughput (bytes per second), and cloud disks price them separately. Four thousand cache-miss reads a second that each touch disk is 4,000 IOPS; a default cloud volume offers 3,000 and charges for more, while a local NVMe drive does hundreds of thousands. A storage estimate that says "12 TB" and stops has answered the cheaper half of the question.

The growth window deserves its own paragraph, because it's the factor with the widest range: 30 days versus 7 years is an 84× difference, and it's a product question with an engineering price tag. Make it concrete. 100M URLs a month, doubling every year. In year 1 you add 1.2 billion records; in year 3, 4.8 billion; in year 5, 19 billion. Those are the annual additions. The cumulative total after five years is 1.2 + 2.4 + 4.8 + 9.6 + 19.2 ≈ 37 billion records, about six times the flat 6 billion the calculation above assumed, so the 12 TB becomes something like 74. The sharding post's rebalancing chapter exists because teams do this math in year four instead of year zero. Growth multiplies every number in this post. Apply it first. And if the retention answer is "forever," the storage math never converges, which is worth raising in the design review.

Section 6 — How fat is the pipe, and does it fit in RAM?

Two questions, one section, because they're both about where the data lives while it's moving and while it's hot.

Bandwidth: response size × requests per second. The URL shortener's redirect: a roughly 500-byte response × 4,000 rps = 2 MB/s = 16 Mbps, a rounding error on any modern link. (Section 2's trap, defused: 16 megabits, which is 2 MB/s × 8.) Now scale it: a thumbnail service serving 50 KB images at 20,000 rps = 1 GB/s = 8 Gbps, which is a real link you have to buy, with real redundancy.

Two corrections to the naive multiplication. Protocol overhead (TCP and HTTP headers, TLS framing) adds something like 5 to 10% to a 50 KB response and can double a 500-byte one, because the headers are bigger than the payload. And bandwidth is not throughput: a single TCP connection can't move more than its window size divided by the round-trip time regardless of the link, so one 4 MB window across a 150 ms ocean tops out near 27 MB/s. Big transfers over long distances need parallel connections or a nearer endpoint, which is one of the CDN post's (#7) reasons for existing.

Bandwidth has a price tag, and it's the egress that bites. (Ingress is data coming into the cloud; egress is data going out to the internet.) Ingress is usually free. Egress runs about $0.05 to $0.09 per GB on AWS and Azure depending on volume tier, and up to about $0.12 on Google Cloud's premium tier. The thumbnail service above moves 1 GB/s, which is about 2,600 TB a month, or somewhere in the range of $130,000 to $210,000 a month at list prices if all of it leaves the cloud directly. Serve the same bytes through the provider's own CDN and the picture changes: the transfer from your origin to the CDN is free, and the CDN's per-gigabyte price at that volume blends to roughly 60% of raw egress, so the bill drops toward $80,000. (Some CDNs charge flat rates rather than per gigabyte, Cloudflare's plans and, since 2025, CloudFront's own monthly plans, which is its own arithmetic.) The estimation didn't just size the pipe; it chose the vendor. When someone proposes a design that multiplies bytes moved, do the egress math before applauding.

Two bandwidth line items that are smaller per byte and forgotten more often: cross-zone traffic (about a cent per GB in each direction on AWS; a Kafka cluster (the log-based message broker) with three replicas spread across three zones sends every byte across a zone boundary twice for replication, and about two-thirds of the produce and consume traffic crosses one too, so 300 TB a month of ingest is closer to 900 TB of cross-zone transfer, billed in both directions, or something like $16,000 to $20,000 a month) and NAT gateway processing (a few cents per GB for traffic from private subnets to the internet). Neither shows up in a diagram. Both show up on the bill.

Memory: does the hot data fit in RAM? This is the single most useful memory question, and it's what makes the caching post's design quantitative. Take the working set (the data that's actually hot) and compare it to the memory you can buy:

• URL shortener: access is Zipfian, meaning a tiny fraction of links takes most of the clicks. Suppose the hottest 20 million keys carry most of the traffic (that's a third of a percent of the 6 billion records). 20M × 500 bytes = 10 GB. One cache node holds that comfortably, plus a replica for failover. The cache design is one shard, not a cluster; the math said so.
• Counter-example: 500 million user sessions × 2 KB = 1 TB of hot session data. That fits in one machine's memory today (cloud instances go to 32 TB), but a single node has one CPU's worth of throughput and a blast radius of "everyone," so it's a sharded cache for reasons of throughput and failure isolation, not because the bytes don't fit.

Size the working set, not the dataset. Nobody caches 6 billion URL mappings. You cache the 20 million that matter. Engineers who size caches for the dataset buy clusters; engineers who size for the working set buy one node. And once you have a hit ratio in mind, finish the sentence: the database sees rps × (1 − hit ratio). At 4,000 rps and a 95% hit ratio, the database handles 200 reads a second; at 99%, 40. That's the number that tells you what the database tier actually needs to be.

Memory has line items beyond the working set that people discover in production. Every open connection holds buffers, so 12,000 in-flight requests at roughly 50 KB each is 600 MB before a single byte of your data. Redis spends 50 to 100 bytes of overhead per key, which on 20 million keys is one to two gigabytes on top of the values. A garbage-collected runtime wants 1.5 to 2 times its live heap as headroom. And a database's most important memory number is whether the indexes fit in its page cache, because that's what decides whether a lookup is a memory read or a disk read. The observability post's (#9) cost-control chapter is the same instinct applied to telemetry: keep the hot window, tier the rest.

Section 7 — Budgets, and the one equation

Sooner or later the design review asks "will it be fast enough?" and "how many workers do I need?", and "I think so" is not an answer. This section is the arithmetic behind both, and it includes the one place where the arithmetic people usually do is subtly off.

Latency budgets. Take your end-to-end budget (say 500 ms for a page load) and slice it per hop. Edge 20 ms, API server 50 ms, cache lookup 2 ms, database 10 ms, three downstream calls at 60 ms each. Add them: 20 + 50 + 2 + 10 + 180 = 262 ms. Under budget, with room. Add a fourth downstream call and a cross-region hop at 100 ms, and you're at 422 ms, one slow dependency away from breaking the promise.

Now the part that's usually said wrong. The numbers above are per-hop p99s, and the natural instruction is "add p99s, never averages." Averages do lie (the observability post, #9, makes that case at length), but summing p99s is a budget heuristic, not a law. Percentiles don't add. If ten sequential hops each have a p99 of 20 ms, the chance that a given request hits the slow 1% on at least one hop is 1 − 0.99¹⁰, about 10%, but one slow hop adds one tail to nine ordinary hops, so "20 ms × 10 = 200 ms" is a pessimistic ceiling rather than the true p99: two hops that are each slow 1% of the time are rarely slow together, and the real end-to-end p99 of the sum is usually well under the sum of p99s (with correlated tails, a shared overloaded dependency, it can exceed it). Parallel fan-out is the reverse problem: if you call N services at once and wait for all of them, every one of them has to be fast for the request to be fast, so the request is slow whenever any leg is slow. Ten parallel legs at p99 give you a p90 result; to hold an end-to-end p99 across ten parallel legs, each leg has to be at roughly p99.9. This is the observation at the center of Dean and Barroso's "The Tail at Scale," and it's why fan-out systems hedge requests (post #2). What to actually do with a budget: sum the per-hop medians to see where time goes, sum the p99s as a conservative ceiling, and for anything that fans out, count the number of legs and reserve the tail accordingly.

The rule the diagram hides: the budget is spent before you draw anything. Every hop you add donates its tail to your p99. When the budget doesn't fit, the fixes are architectural (parallelize the downstream calls, move work to the edge, cache the slow hop), not "optimize the code 10%." Estimation tells you the budget doesn't fit on paper, which is infinitely cheaper than learning it in production.

Little's law: concurrency = throughput × latency. One equation, endless uses. If requests arrive at λ per second and each spends W seconds in the system, then on average L = λ × W requests are in the system at once. That's the law, in full. John Little proved it in 1961, and it holds for any stable system regardless of what's inside.

The clean worked example is sizing a database connection pool. 4,000 reads a second, each taking 5 ms against the database: L = 4,000 × 0.005 = 20 concurrent connections. A pool of 50 has comfortable headroom; a pool of 10 will queue and melt. Another: an ingest worker handling 120,000 events a second at 100 ms each needs 12,000 concurrent slots, which tells you immediately that one-thread-per-event won't work and you need batching or asynchronous I/O (handling many in-flight operations per thread instead of one). Little's law turns "how many threads, connections, workers?" from a guess into arithmetic.

It also runs backward: 2,000 in-flight requests each taking 200 ms means you're doing 10,000 rps, which is useful when all you can observe is queue depth. It sizes queues: a worker pool draining 10,000 messages a second at 200 ms each has 2,000 in flight, so the buffer in front of the workers must hold bursts above that without dropping. Size the buffer from Little's law, not from a default config value. And it exposes lies: a vendor claiming a million rps per node with 10 ms processing is claiming 10,000 concurrent operations per node, which is possible, and now you know what to ask about.

The queueing curve behind the 60% rule. Section 9 will tell you to run at no more than 60% utilization at peak, and that number deserves a reason rather than folklore. In the simplest queueing model (one server, random arrivals, random service times; queueing theorists call it M/M/1), the time a request spends waiting-plus-being-served is the service time divided by (1 − utilization); with steadier service times the queueing part roughly halves, but the shape is the same. At 50% utilization, requests take twice as long as the bare service time. At 60%, two and a half times. At 80%, five times. At 90%, ten. At 100%, the wait is unbounded and the queue grows forever. Real traffic is burstier than the model, which moves the knee left. That curve is the whole argument for headroom: the last 30 to 40% of a machine's capacity isn't spare, it's the part where latency goes vertical.

Concurrency has one more physical limit that the arithmetic above hides. Twelve thousand in-flight requests means twelve thousand open sockets, and sockets run into things that aren't CPU: a process's file-descriptor limit, the roughly 28,000 ephemeral ports one machine has per destination, and connection-tracking tables in NATs and load balancers (the load balancer post, #11, has that list). These limits get hit before the CPU does, and they're a capacity dimension in their own right.

Section 8 — The traps

Estimation fails in predictable ways. Here are the five big ones, in the order I've watched them bite, and four small ones that hide behind them.

1. Bits versus bytes. Covered in Section 2, repeated here because it's the number one killer. "10 Gbps link, 10 GB file, so one second." No: eight seconds. Say the units.

2. Forgetting the replication factor. The raw data is never the stored data. ×3 for replicated datastores, ×2 for mirrored disks (RAID 1), ×1.3 to 1.5 for index overhead. Section 5's team bought a quarter of the disks they needed. Estimate the database, not the data. (And the reverse: don't apply it to object storage, which replicates itself and bills logical bytes.)

3. Average versus peak. Provisioning for the average is provisioning for the outage. The daily curve peaks at two to three times; launches and events hit ten. Compute the average, multiply by the peak factor, provision for that.

4. Ignoring overhead. Indexes, metadata, filesystem slack, serialization bloat (JSON is about twice the size of a packed binary format), write amplification in log-structured engines. The working rule: add 30 to 50% on top of every storage number and 10 to 100% on every bandwidth number depending on payload size, and write down that you did.

5. Growth blindness. "100M URLs a month" is true now. Designs live for years. Ask what the number is in three years at 2× annual growth: an 8× multiplier hiding in the requirements doc. If nobody can answer the growth question, pick 10× headroom and say so.

And the small ones: counting only the requests at the edge and forgetting the fan-out inside (Section 4); letting 1,000-versus-1,024 compound to 10% at the terabyte scale and then wondering why the disk is full; estimating replication storage but not replication traffic, which is writes × replicas across a zone boundary every second (Section 6); and trusting "the average record is 500 bytes" when the distribution has a fat tail of 50 KB records that own half the storage.

Three habits dodge all of them:

• State assumptions, in writing. "Assuming 500-byte records, 3× replication, 2× peak factor." Written down, they can be challenged. Kept in your head, they're invisible load-bearing guesses.
• Round aggressively. One significant figure. If the answer changes when you round, the answer was never robust; redesign, don't refine.
• Sanity-check against known systems. A single Postgres does thousands of transactions a second and tens of thousands of simple inserts. A Redis node does 100,000-plus reads a second. A Kafka cluster does millions of events a second. If your estimate says one Postgres will handle 500,000 writes a second, the estimate is wrong, not the database.

An estimate is a bet with the assumptions written on it. Write them down, or you're not estimating. You're guessing.

Anchor systems, the capacities you memorize so that absurd results get caught instantly:

These rot slowly as hardware improves, but the orders of magnitude hold for years. If your estimate disagrees with an anchor by 10×, either the estimate is wrong or the anchor is stale, and either way you've learned something before buying anything.

Section 9 — Going deep: cost, capacity, and knowing when to stop

The arithmetic is done. What remains is the part estimates can't do for you: what it costs, how much headroom to buy, how to check the estimate against reality, and how much precision each question deserves.

Cost: dollars per million requests. The unit that makes infrastructure comparable. Take the monthly bill for a component, divide by the millions of requests it serves:

• 4 servers of the c6i.xlarge class at \(0.17/hour ≈ \)124 a month each ≈ $500 a month, serving 10M requests a month → $50 per million requests.
• A managed API gateway charges $1.00 to $3.50 per million (AWS's HTTP API and REST API tiers, respectively). At 10M a month that's $10 to $35, so managed wins, because you're not paying for idle servers.
• At 10 billion requests a month the math flips hard. The gateway's tiered pricing works out to roughly $10,000 to $25,000 a month. Ten billion a month is about 3,900 requests a second on average and maybe 12,000 at peak, which four to eight of those same servers handle for $500 to $1,000 plus a load balancer. Self-hosted is something like 25 times cheaper at that volume.

That's the real use of cost estimation: it tells you where the managed-versus-self-hosted line sits, and the line moves with volume. Count what's on both sides of it, though. A managed queue at $0.40 per million requests (SQS's list price, and a message costs at least three requests: send, receive, delete) is around $120 a month at 100 million messages and nobody thinks about it; at 10 billion it's around \(12,000, and a three-broker Kafka cluster is maybe \)1,500 in instances. Instances aren't the whole cost. The cluster needs someone to upgrade it, monitor it, and get paged for it, and an engineer-week a quarter is more than the difference. The per-million arithmetic tells you when the conversation is worth having. It doesn't end the conversation. Compute the unit cost before the volume does it for you, and then add the people.

Two pricing facts move the line further: reserved instances cut the $0.17 to about $0.11, and spot instances to about $0.08, which shifts the self-hosted side by 1.5 to 2×; the managed side has volume tiers that shift it back. Neither changes the shape, just where the crossover lands.

Capacity: headroom, and how much. Two rules of thumb that have survived decades, now with the reasons attached:

• Run at or below 60% utilization at peak. Section 7's queueing curve is why: at 60% a request already takes 2.5 times its bare service time, and the curve goes vertical past 80%. The remaining 40% is for the spike you didn't predict, the deploy that's mid-rollout, the neighbor that's noisy. A box at 95% has no slack; a box at 60% has a future.
• N+2, not N+1. If N boxes handle peak at 60%, the naive rule says buy N+1 so one can die. Google's SRE book argues for N+2: one box out for the rolling deploy that's always in progress, and one for the failure that happens during the deploy. Peak needs 100,000 rps; one box serves 10,000 of simple logic, so 6,000 usable at 60%. Seventeen boxes give 102,000 usable; buy nineteen. The extra boxes are the cheapest insurance in the building.
• Zones and regions. If the fleet is spread across k availability zones and you want to survive losing one, each zone can run at no more than (k − 1)/k of capacity: across three zones, that's 1.5× peak provisioned in total. Surviving a region failure with a warm second region is 2× peak. Say which failure you're buying insurance against, because the price doubles between them.

Sizing compute from first principles. When you don't have an anchor, derive one: boxes = peak rps × CPU-milliseconds per request ÷ (cores per box × 1,000 × 0.6). If a request costs 2 ms of CPU and peak is 350,000 rps, that's 700 core-seconds every second, about 1,170 cores at 60% utilization, or roughly 290 four-core boxes. The CPU-per-request number is the one you measure (see load testing, below), and it's the one that batching and asynchronous I/O attack: cut it from 2 ms to 0.5 and the fleet drops to about 75 boxes.

Load testing: an estimate is a hypothesis. Everything above produces a number to check, not to believe. Take one box, load it to 1.5× its projected share of peak, and find two things: the throughput where latency leaves the flat part of Section 7's curve (that's the box's real capacity), and the CPU-milliseconds per request (that's the input to the sizing formula). One afternoon with a load generator replaces a page of assumptions. Then re-check when the code changes materially, because CPU per request drifts.

Growth, as a formula. Traffic at time t is roughly today's traffic × (1 + g)ᵗ for annual growth rate g. The rule of 70 gives the doubling time: 70 divided by the growth percentage, so 40% annual growth doubles in about two years (the rule says 21 months; the exact answer is 25, and the rule is only meant for small rates). Plan capacity to a 12-to-18-month horizon, and set a trigger (when sustained peak passes 50% of provisioned capacity, order more) that accounts for lead time, whether that's a week for cloud quota or a quarter for hardware.

Estimating from the other direction. Sometimes the constraint is the budget and the question is how much scale it buys, and the pipeline inverts cleanly. "$10,000 a month for egress at $0.05/GB is 200 TB a month, which is about 77 MB/s, which at 4 KB per response is roughly 20,000 requests a second." Now the product conversation has a number in it.

When precision matters, and when it doesn't. This is the judgment call, and it's a spectrum:

• Choosing the architecture: one significant figure. "Is it 10 rps or 10,000?" decides cache-versus-database. More digits add nothing.
• Buying hardware or signing contracts: ±30%. Get the peak factor and the growth assumptions right; the rest is margin.
• Billing, SLAs, and promises to customers: exact. "99.9% over 30 days" is 43.2 minutes; compute it precisely, because the contract is precise. (An SLA is a service-level agreement: the contractual version of a promise about uptime or latency, with penalties attached.)

The meta-rule: estimate to the precision of the decision. A choice between two architectures needs one digit. A purchase order needs two. Most estimation arguments are people computing six digits for a one-digit decision. Stop when the next digit can't change the answer.

One more habit: re-estimate on a schedule, not on an incident. The numbers decay. Traffic grows, ratios shift when a new feature lands, retention policies change. The teams that do this well redo the napkin math quarterly, fifteen minutes, same pipeline, and catch the quarter where the cache stops fitting in RAM before the pager does. Estimation isn't a design-phase activity. It's a habit.

Now the full pipeline, one pass, on an analytics ingest: 10 billion events a day, 1 KB per event, 30-day retention, events landing in Kafka and settling into object storage:

• Traffic: 10¹⁰ ÷ 86,400 ≈ 120,000 rps average (100,000 by the divide-by-10⁵ rule; either is fine at this precision). Peak 3× → 350,000 rps to provision for.
• Storage: 10B × 1 KB = 10 TB a day × 30 days = 300 TB of raw events. This lands in object storage, so no replication factor and no index overhead; compressed as Parquet with zstd, more like 30 to 60 TB on disk. Had it been a replicated database, the same math would have read 300 TB × 3 × 1.4 ≈ 1.3 PB, and the difference between those two numbers is the point of Section 5's second warning.
• Bandwidth: 120,000 × 1 KB = 120 MB/s ≈ 1 Gbps of ingress on average, about 3 Gbps at peak. Kafka with three replicas across three zones adds tens of thousands of dollars a month of cross-zone traffic that no diagram shows.
• Memory: the last-hour hot window for live dashboards is 120,000 × 3,600 × 1 KB = 432 GB. That fits on one large machine, but one machine is one CPU's worth of query throughput and one failure domain, so it's a sharded hot store or a pre-aggregated one.
• Little's law: 120,000 rps × 100 ms of processing = 12,000 concurrent. One-thread-per-event is dead; batching or async I/O is required.
• Compute: 350,000 peak × 2 ms of CPU per event = 700 core-seconds a second → about 1,170 cores at 60% → roughly 290 four-core boxes, about $36,000 a month at on-demand prices. This is the line that dominates. Batching to cut CPU per event is worth more than any storage optimization.
• Storage cost: 300 TB uncompressed at $0.023/GB-month ≈ $7,000 a month; compressed, closer to $1,000. Small next to compute.
• Capacity: provision for 350,000 at 60% → design for about 600,000 rps; N+2 across the ingest fleet; 1.5× across three zones if one zone's loss must be survivable.
• Sanity check: Kafka clusters routinely do millions of events a second, so 350,000 at peak is big but ordinary. One Postgres would not survive it. The guardrails hold.

Ten numbers, five minutes, and the architecture is sketched: batched ingest, a sharded hot store, object storage for the cold 300 TB, and a compute bill that dwarfs the storage bill, which tells you where the optimization effort goes. That is the post in one pass, and it's exactly what Step 2 did for the URL shortener, one level up.

Do the math first, distilled

The latency table (one significant figure):

The unit toolkit:

The formula sheet:

The traps: bits vs bytes · forgotten replication factor (and replication applied to S3) · average vs peak · missing overhead · growth blindness · counting only the edge · KB/KiB at scale · replication traffic · the fat-tailed record.

The technique: state assumptions · round to one significant figure · sanity-check against known systems · load-test the number you got · stop when the next digit can't change the answer.

Further reading

• Jeff Dean's latency numbers, Jonas Bonér's gist. The cheat sheet in its most-linked form, with credit to Peter Norvig's original.
• The humanized scale (1 ns = 1 second). The version that makes it stick.
• Peter Norvig: Teach Yourself Programming in Ten Years. Where the original table lives, near the bottom.
• Little's law. The formal statement and the 1961 proof.
• Dean & Barroso: The Tail at Scale (CACM 2013). Why percentiles don't add and what fan-out does to tails; behind Section 7.
• Google SRE Book: The Production Environment at Google. The N+2 argument.
• Designing Data-Intensive Applications by Martin Kleppmann. Chapters 1, 3, 5, and 6 put every number in this post into architectural context.

Where you'll meet this

Every design interview's first five minutes. Every capacity review, every "can we handle Black Friday" meeting, every bill-shock postmortem. It decided the caching post (#1, the 100:1 ratio), sized the key blocks in the Snowflake Problem (#4), placed the replicas in Copies of the Truth (#5), set the alert thresholds in the observability post (#9), and gave the load balancer post (#11) its headroom rule. The queueing curve in Section 7 is the reason the resilience post (#2) treats utilization headroom as a feature rather than waste. Estimation is the first skill and the last check: the thing you do before the design and the thing you redo when the design is under stress. Next up is the capstone (#14), where all of this gets forty-five minutes and a whiteboard.

Do the math first. The boxes can wait.

Keep exploring: the Core Concepts series

Every post in the series stands alone. Read them in any order.

• From Laptop to Billions of Clicks: Designing a URL Shortener in 11 Steps — the anchor: one design, every concept under load.
• #1 The 100:1 Superpower: Caching — the fastest request is the one you never make.
• #2 The Blast Radius: Surviving the Day Your Dependencies Fail — staying up when everything you depend on goes down.
• #3 Divide and Conquer: Sharding and Partitioning — splitting one database into many without losing your mind.
• #4 The Snowflake Problem: Unique IDs at Scale — naming things when millions are born every second.
• #5 Copies of the Truth: Replication — keeping copies of your data that actually agree.
• #6 Do No Harm Twice: Idempotency — making "just retry it" safe.
• #7 The Copy at the Doorstep: CDNs and Edge Computing — serving from next door instead of across the ocean.
• #8 The Bouncer's Math: Rate Limiting — saying no politely, at scale.
• #9 What Broke at 3 AM: Observability, p99, and Useful Alerts — knowing what's wrong before your users tell you.
• #10 Do It Later, On Purpose: Async Processing and Queues — the work the user doesn't have to wait for.
• #11 The Traffic Cop: Load Balancing — the box in every diagram nobody explains.
• #12 Assume They're Already Knocking: Security and Abuse at Scale — designing for the users who are designing against you.
• #13 Do the Math First: Estimation for System Design — the two minutes of arithmetic that choose the architecture. (this post)
• #14 The Whiteboard Playbook: Taking On Any System Design Challenge — the whole series in forty-five minutes.

Blueprints of Scale — Core Concepts #13. If this helped, the best thanks is a share with someone who's learning.

Here's the idea in full: most of the work your system does doesn't need to happen while the user is waiting. The charge needs to go through now. The confirmation page needs to render now. But the receipt email, the analytics event, the search index update, the thumbnail render, the fraud check, the warehouse notification: none of those need the user's attention, and doing them inline is what makes the spinner spin. Async processing is the discipline of separating "must happen now" from "must happen eventually," and building the machinery that guarantees "eventually." That machinery is the queue, and it's one of the highest-leverage ideas in backend engineering: it makes systems faster for the user, tougher under spikes, and simpler to reason about at scale. It also has failure modes that will page you if you skip the boring parts. This post covers both.

Here's what's covered: the seven-second checkout and the one question that fixes it; what "async" actually means (it's about whose time you're spending); the queue as a shock absorber, and the shapes queues come in (point-to-point, pub/sub fan-out, and the log); why delivery is at-least-once, why your consumers must be idempotent, and where the at-most-once switch actually lives; the dual-write problem, the transactional outbox, and the relay's own failure modes; workers: parallelism and its ceiling, ordering versus throughput, poison messages, dead-letter queues, priorities, batching, big payloads, and noisy tenants; delayed and scheduled jobs, distributed locks, and retries with backoff; backpressure, properly defined, and the three things you can do when producers outrun consumers; and the principal-level toolkit: choreography versus orchestration and the engines that do it, the lag arithmetic, event-driven pitfalls (ordering, replay, schema evolution), what async does to the user experience, tracing across queues, and when not to go async at all.

Sections 1 and 2 assume nothing. Sections 3 through 8 are the machinery every backend engineer meets in production. Section 9 is the judgment. The cheat sheet is at the end under Do It Later, On Purpose, distilled, and every diagram is also described in the text around it.

Section 1 — The seven-second checkout

An online store's "Place Order" button takes seven seconds to respond. Every time, not just sometimes. The team profiles it and finds the checkout handler doing its work like a diligent clerk with no sense of priority: charge the card (1.2 seconds), reserve the inventory (0.8), send the confirmation email (1.5), send the receipt email (1.5), update the analytics pipeline (1.0), add the loyalty points (1.2). Total: 7.2 seconds of spinner. Every step correct. Every step necessary. Every step happening while the user stares at the screen.

One picture, and it's the whole problem: a chain where every link is someone else's latency. And it gets worse than slow. Each of those six steps is a dependency (the email service, the analytics pipeline, the loyalty system), which means the checkout now fails when any of them fails. The loyalty service has a bad deploy? Checkout is down. The email provider is slow today? Every customer waits.

Synchronous work doesn't just add latency. It multiplies your blast radius.

The resilience post (#2) called this out: every inline dependency is a way to be down.

Now the question this entire post is built on: which of those six steps does the user actually need before they see "order placed"? The charge, obviously; you can't confirm an order you haven't taken money for. The inventory reservation, yes; you can't sell what you don't have. The other four? The user will never know whether the confirmation email sent in 400 milliseconds or 4 minutes. The analytics pipeline definitely doesn't care. The loyalty points can land whenever. Four of the six steps are hostages to the user's attention for no reason.

The fix the team shipped: the request handler does the charge and the inventory reservation, the two things the answer depends on, drops four messages onto a queue ("send confirmation email," "send receipt," "record analytics," "add loyalty points"), and returns. Checkout went from 7.2 seconds to 2.0. The emails still send. The analytics still update. The loyalty points still land. They just happen after the user has their answer. (The remaining two seconds are the payment provider's and the inventory system's, and they're a different kind of problem: those are real synchronous dependencies, and making them faster is the estimation post's (#13) and the resilience post's (#2) business, not this one's.)

Nothing got faster. The user just stopped waiting for it.

There's a lesson hiding in that last line: latency has an audience. Work the user waits for has a budget measured in hundreds of milliseconds. Work nobody waits for has a budget measured in minutes, and a much bigger budget means much cheaper engineering. The rest of this post is the machinery that makes "later" reliable.

Section 2 — The one question: does the user need the result now?

Every piece of work your system does, run it through one question: does the user need the result now? Not "is it important" (the receipt email is important). Not "is it part of the flow" (the analytics event is part of the flow). The question is strictly about the response: does the answer I'm about to send depend on this work being done?

The "yes" bucket, the truly synchronous work, is smaller than most codebases suggest: authentication (you can't serve the request without knowing who it's for), the decision the response depends on (is the seat available? did the charge clear? is the username taken?), and reads (fetching data to display is inherently synchronous; the page needs it). The "no" bucket is everything else, and it's enormous: notifications of every kind, analytics and metrics, search index updates, thumbnail and preview rendering, cache warming, webhooks to third parties, reports and exports, fraud and abuse scoring that doesn't gate the response, data warehouse syncs.

A week later, the same team from Section 1 did the exercise properly, to check the fix they'd shipped in a hurry. They listed every side effect in the checkout flow and asked the question about each one, in a room, with the product manager present. Charge: yes. Inventory: yes. Confirmation email: no. Receipt: no. Analytics: no. Loyalty: no. Fraud check, which the first pass had missed entirely: interesting. The fraud score didn't gate the order, it gated the shipment, which happened hours later. So: no. Seven items, two yeses. The exercise took twenty minutes, confirmed the five seconds they'd already saved, and found one more item to move. Most "slow endpoints" are slow the same way: not because any step is slow, but because steps the user doesn't need are spending the user's time.

Underneath is a useful mental model: every request has a latency budget, and synchronous work spends it. The classic thresholds from usability research (Jakob Nielsen's, from 1993, and still cited) are that 100 milliseconds feels instant, one second keeps the user's flow of thought intact, and ten seconds is the limit of attention. Most teams budget a few hundred milliseconds for an interactive response, and every inline dependency, every email send, every analytics call, draws from the same small account. Async work draws from a different account entirely, one measured in minutes, with effectively infinite funds.

One caveat before the machinery: "later" has to actually happen, reliably and observably, and that's Sections 3 through 8. "Fire and forget" without the machinery is just "forget." The question tells you what to defer; the queue tells you how to keep the promise.

Async is about moving work to the budget where it's cheap, not about making it faster.

Section 3 — The queue as a shock absorber

A queue is embarrassingly simple, which is why it works. Three roles. Producers put messages on the queue ("send this email," "process this image") and move on; they don't know or care who handles the message or when. The broker holds the messages until someone takes them, and a properly configured broker holds them durably, on disk and replicated, so a crashed broker doesn't lose them (that "properly configured" is doing work: RabbitMQ's transient queues, a Kafka topic with a replication factor of one, or a Redis list with default persistence are not durable, whatever the marketing says). Consumers, also called workers, pull messages off, do the work, and acknowledge completion (an "ack" is the consumer telling the broker "done, you can delete this"). The producer and the consumer never meet. They don't even have to exist at the same time: the producer can be long gone before the consumer wakes up. The queue decouples work in time and in knowledge, and that decoupling is what absorbs shocks.

Take the concert ticket on-sale: a hundred times normal traffic at 10 AM sharp, 100,000 messages a minute for ten minutes, a million messages. Without a queue, that spike hits your workers directly: they saturate, requests time out, the on-sale collapses, and angry fans trend on social media. With a queue, the spike hits the broker, which just holds messages. It's very good at holding messages; that's its whole job. Ten workers at 2,000 a minute each chew through the pile at their sustainable pace, and the last confirmation email goes out about fifty minutes later. Nobody at the concert cares that their email arrived at 10:50 instead of 10:05. The queue converted a capacity emergency into a latency bill, and for background work, latency is the cheapest currency there is.

Queues also act as bulkheads, the resilience post's term for walls that stop one failure from sinking the ship. If the email service goes down for an hour, the messages wait in the queue instead of failing the checkout. When the service recovers, the workers drain the backlog. The producer never knew there was an outage. Without the queue, a downstream outage is your outage. With it, it's a delay.

Now, "queue" covers three different shapes, and picking wrong causes real pain:

Point-to-point (SQS standard and FIFO queues, RabbitMQ's classic and quorum queues, Azure Service Bus queues): a message is delivered to one consumer, and once acknowledged, it's gone. This is the work-distribution shape, sometimes called competing consumers: a pool of workers pulling tasks, each task done once (well, at-least-once; Section 4). Use it when the work is tasks: send emails, render thumbnails, process uploads. Parallelism is simple: add workers.

Publish/subscribe (SNS fanning out to SQS queues, RabbitMQ exchanges with several bound queues, Service Bus topics with subscriptions, Google Pub/Sub): one message, many readers, each getting its own copy. "Order placed" goes to the warehouse's queue, the email service's queue, and the analytics queue at once, and each consumes at its own pace. This is the fan-out shape, and it's how you get "many readers" without a log. What it doesn't give you, by default, is history: once each subscriber has consumed its copy, the message is gone (some, like Google Pub/Sub, can retain and replay if you turn it on).

Log-based (Kafka, Redpanda, Pulsar, Kinesis, and RabbitMQ's newer streams): the broker is an append-only log. Messages are never deleted on read; they're kept for a retention period (Kafka's default is seven days, configurable per topic), and each consumer group tracks its own position (its offset) in the log independently. Five different systems can each read the same events at their own pace, and a new consumer can rewind and replay history from last Tuesday. This is the event shape: "order placed" is a fact that the warehouse, the analytics team, and the fraud system all want to hear about, separately, and that a system built next year will want to hear about retroactively. The price is that parallelism is capped by the number of partitions the topic was created with (each partition is read by one consumer in a group; extra consumers sit idle), changing the partition count is a migration, and on the classic protocol every time a consumer joins or leaves the group, the group pauses to rebalance (Kafka 4.0's incremental protocol removes most of that pause).

The way teams learn the difference is by choosing point-to-point for events. Order events go into a classic RabbitMQ queue; then the analytics team wants the same events, but they've been consumed and deleted; then someone wants to reprocess last month after a bug fix, and the messages are gone. The migration that follows moves the event streams to a log and keeps the queue for the actual tasks (send the email, charge the card), and everyone stops fighting the tool.

As a default: tasks go point-to-point, fan-out goes pub/sub, facts go on the log. It's a default, not a law; Kafka gets used for task queues at plenty of companies and now has share groups built for it. What matters is that when someone proposes using one shape for another's job, you ask where the replay button is, or where the second reader gets its copy.

Two properties that belong in every queue's design and rarely make it in:

• Expiry. Messages have a lifespan. SQS keeps a message for four days by default and fourteen at most; RabbitMQ has per-message and per-queue TTLs; Kafka has retention. A backlog that takes five days to drain silently expires the work at the back of the line, and a dead-letter queue (Section 6) with a shorter retention than its source queue loses the very messages it was meant to preserve. Set both deliberately.
• Cost and latency. The queue hop itself costs milliseconds and money. Hosted queues bill per request (SQS charges per million API calls, which is why batching sends and receives and using long polling matter), and a self-run broker bills in engineer-hours. Neither is a reason not to use a queue. Both are reasons to know the number.

Section 4 — At-least-once: the delivery guarantee you actually get

Exactly-once delivery is impossible. The idempotency post (#6) walked through the argument in its Section 4, and it fits in two sentences: the broker can't know the consumer finished unless the consumer says so, and the acknowledgment itself can be lost. So this section is about what you build instead of chasing the impossible, and about a distinction the vendors blur: delivery is impossible to make exactly-once; processing is not, as long as the record of "I did this" and the effect itself are committed together.

Queues make three promises about delivery:

• At-most-once: the message is delivered zero or one times. Fast, simple, and it can lose your message: the broker hands it to a consumer, the consumer crashes before processing, and nobody ever knows. Use it for metrics and sampling, where a lost data point is noise.
• At-least-once: the message is delivered one or more times. Nothing is lost, but duplicates happen. This is what every serious queue promises by default, because it's the only reliable promise available.
• Exactly-once: delivered exactly one time. See above: not available at the boundary between the broker and your code. Any vendor claiming it means exactly-once within a scope: Kafka's transactions give exactly-once processing for pipelines that read from Kafka and write back to Kafka; SQS FIFO deduplicates sends that carry the same deduplication ID within a five-minute window. Both are real and useful. Neither reaches the code that sends the email.

Where the switch between at-most-once and at-least-once actually lives is worth knowing, because it's a configuration setting rather than a product choice. It's the ack. A consumer that acknowledges before doing the work (auto-ack in RabbitMQ, committing the Kafka offset as soon as the batch arrives) has chosen at-most-once: crash after the ack and the message is gone. A consumer that acknowledges after the work (manual ack, commit after processing) has chosen at-least-once: crash before the ack and the message comes back. Choose the second and build for duplicates.

Six steps, and that is the entire contract. The consumer did the work, the ack got lost, the broker redelivered (correctly, by its contract), and the consumer's dedup check turned the duplicate into a no-op. The reliability lives at the edge, not in the pipe. This is the idempotency post's central result in work clothes: at-least-once delivery plus an idempotent consumer (one that can safely handle the same message twice) equals effectively-once processing.

The welcome-email worker with no dedup is the standard cautionary tale. It sends the email, then crashes before acknowledging; a deploy rolled mid-processing. The broker redelivers. It sends the email again, crashes again (same deploy, same bad luck). One new user gets the welcome email three times, and the third copy's "We're so glad you're here!" reads as mockery. The fix is a processed_messages(message_id PRIMARY KEY) table and a check before sending. That table has a name, the inbox (the receiving-side twin of Section 5's outbox), and it has one rule that the naive version breaks: the check and the business write have to happen in one database transaction, so that "insert the message ID, then send, then crash before commit" rolls back cleanly and "insert the ID and commit, then crash before sending" can't happen. Check-then-act with a gap between them is the race the idempotency post's Section 5 is about. Duplicates are rare until the day they're a flood, and the inbox is flood insurance.

The mechanics you'll configure: the visibility timeout is SQS's term (Google Pub/Sub calls it the ack deadline) for how long a message stays hidden from other consumers after one takes it, 30 seconds by default, up to 12 hours. Finish and ack within the window, and the message is deleted. Crash or run long, and the message becomes visible again, redelivered by design. Set the timeout longer than your slowest legitimate processing time, or extend it with a heartbeat call while the work is running, or slow messages get processed twice while the first attempt is still running. RabbitMQ works differently: an unacknowledged message is redelivered when the consumer's connection or channel closes, and a consumer that holds a delivery unacknowledged past the acknowledgement timeout (30 minutes by default) has its channel closed by the broker, which requeues everything on it. Same outcome, different trigger.

One more interaction, since it bites the ordered queues: if a message in a FIFO message group or a Kafka partition keeps failing and being redelivered, everything behind it in that group waits. Ordering plus redelivery means a stuck message stalls its whole key, which is Section 6's poison-message problem with a narrower blast radius.

So the default for everything that matters: at-least-once, acked after the work, with an idempotent consumer. At-most-once is for telemetry you can afford to lose. And when a vendor's "exactly-once" marketing reaches your inbox, translate it: exactly-once inside their boundary, at-least-once delivery to your code, and the dedup still yours.

Section 5 — The dual-write problem and the transactional outbox

Your handler does two things: writes the order to the database, and publishes "order placed" to the queue. Two writes, two systems, no shared transaction. Now crash between them. Two ways to lose:

• The lost event: the database commit lands, the process dies before publishing. The order exists; the warehouse never hears about it. Nobody ships anything, and nobody knows.
• The phantom event: the publish lands, the database transaction rolls back. The warehouse ships an order that doesn't exist. Someone gets a free television.

That diagram is the fix: the transactional outbox. Instead of publishing to the queue directly, the handler inserts the event into an outbox table in the same database transaction as the order itself. The transaction is atomic (both rows commit or neither does), so the lost event and the phantom event are both impossible by construction. A separate relay process reads new outbox rows and publishes them to the queue, then marks them sent (or deletes them). If the relay crashes mid-publish, it retries on restart, and the event might publish twice, which is exactly what Section 4's idempotent consumer is for. The outbox doesn't eliminate duplicates. It eliminates the two failure modes duplicates are preferable to.

The naive dual-write runs fine for a year. Then a deploy rolls during peak and kills a dozen handlers between the two writes. Twelve orders exist that the warehouse never saw, and customer support finds them three days later, by hand, in a spreadsheet. The postmortem's fix is the outbox table and a fifty-line relay, and the engineer's comment in the code review is the one to remember: "We had a distributed transaction and didn't know it. Now we don't."

The relay is a small program with its own failure modes, and they're worth listing because they're where outbox implementations go wrong in their second year:

• It has to be a singleton, or coordinated. Two relays reading the same outbox publish everything twice (tolerable) and, worse, out of order (not tolerable if order matters). Either elect one leader (Section 7 has the mechanics) or have several relays claim rows with SELECT … FOR UPDATE SKIP LOCKED so each row has exactly one publisher.
• Its poll interval is your latency. A relay that polls every second adds up to a second before the event exists. That's fine for emails and not for fraud checks; know which you have.
• The outbox grows. Sent rows have to be deleted or archived, or the table becomes the largest one in the database and the relay's scan slows down.
• It can fall behind, silently. Relay lag (the age of the oldest unsent row) is its own metric with its own alert. An outbox with ten thousand unsent rows is a system that thinks it's publishing and isn't.

The industrial version, for when you outgrow the polling relay: change data capture (CDC), a tool like Debezium reading the database's transaction log (Postgres's write-ahead log, MySQL's binlog) and publishing row changes as events. Same guarantee (the log is the transaction's truth), no polling, more infrastructure. Debezium's outbox recipe even deletes the outbox row in the same transaction that inserts it, since CDC sees the insert regardless. Start with the outbox table; graduate to CDC when the relay becomes a bottleneck you can measure.

Two patterns sit next to the outbox and get confused with it. The first is the reason you need it at all: there's no practical way to run one transaction across your database and your broker (the two-phase commit protocols that try, XA and friends, are slow, block on failures, and most brokers don't support them), so the outbox gives you the same effect by putting both writes in the database and letting the relay carry one of them out. The second is the saga: when a business operation spans several services (charge, reserve, ship), each step is a local transaction and each has a compensating action that undoes it (refund, release, cancel), so a failure at step three runs the compensations for steps two and one. Sagas are Section 9's orchestration problem, and the sharding post (#3, Section 7) and the idempotency post (#6, Section 6) both cover why the compensations must be idempotent. And a footnote for the pattern people ask about: event sourcing stores the events as the source of truth and derives the current state by replaying them, often paired with CQRS (separate read and write models). It's powerful for audit-heavy domains and a poor fit for ordinary CRUD applications, where it multiplies complexity for no one's benefit.

If the event is best-effort (analytics, metrics), the dual-write is fine and the outbox is ceremony. If the event drives money or fulfillment, the outbox is the difference between "the warehouse was told" and "we hope the warehouse was told."

Section 6 — Workers: parallelism, ordering, and poison

Workers are the unglamorous half of the system, and they're where most async outages actually live. The happy path is simple: add more workers, chew through messages faster. Parallelism is nearly linear until the work itself bottlenecks (the database the workers write to, the third-party API they call) or, on a log, until you run out of partitions. The queue scales easily; the things the workers touch do not. Size the worker pool against the downstream, not the queue.

Then the tension that shapes every worker design: ordering versus throughput. One queue with ten workers means ten messages processed concurrently: fast, and in no particular order. For sending emails, that's perfect. But some work has to happen in sequence: you can't process "order cancelled" before "order placed" for the same order, and you can't apply "set address to B" before "set address to A" and end up correct. The standard answer is partitioning by key: all messages for the same order go to the same partition (Kafka) or message group (SQS FIFO), each partition is consumed by one worker at a time, and order is preserved within the key while different keys still process in parallel. (This is the sharding post's idea of splitting by a key, applied to a stream instead of a table.) You don't choose between ordering and throughput globally. You choose per key.

The right side of that diagram is the section's real subject: poison messages. Some message (malformed, unexpected, a 2 GB "image" that isn't an image) crashes the worker every single time. The worker picks it up, dies, the message becomes visible again, another worker picks it up, dies. In an ordered queue, everything behind it in that partition or group stops; that's head-of-line blocking, the async equivalent of one broken-down car stopping the highway. In an unordered queue the damage is subtler: each redelivery costs one worker one crash-and-restart cycle, so throughput drops by however many workers are busy dying at any moment, and if the crash is an out-of-memory kill (OOM: the operating system terminates a process that has used more memory than it's allowed), the restart takes long enough that a pool can spend most of its time restarting.

The thumbnail service version: uploads processed happily for months, until a user uploads a "photo" that's actually a 2 GB corrupted TIFF. The thumbnail worker loads it into memory, gets OOM-killed, restarts, picks up the same message, gets OOM-killed again. Every worker in the pool takes its turn. Thumbnails stop company-wide, for all users, because of one file and a crash loop. The fix has two parts, and you want both: a retry cap (try three to five times, then stop; a message that fails five times isn't unlucky, it's poison), and a dead-letter queue, the DLQ, where exhausted messages go to wait. In SQS that's a redrive policy with a maxReceiveCount; in RabbitMQ it's a dead-letter exchange with a delivery limit on quorum queues. The DLQ is the morgue with a purpose: the poison is quarantined instead of blocking the living, a human can inspect it, and after a fix the messages can be replayed.

Every queue that carries work you can't afford to lose gets a DLQ. A queue without one is a system where one bad message is a company-wide outage.

The two exceptions to "every queue": an at-most-once telemetry queue where dropping is the design, and a FIFO queue where the vendor warns (SQS does) that dead-lettering a message breaks the ordering guarantee for its group, so you decide deliberately between "stall the key" and "skip the message." And a rule for the DLQ itself: give it a longer retention than its source, because a standard SQS queue counts a message's age from its original send, and a DLQ with the same four-day retention can expire a message the moment it arrives (FIFO queues reset the clock on the move).

Several more worker mechanics that decide whether a system is boring or exciting:

Prefetch. A worker shouldn't grab a hundred messages at once, because a crashed worker returns all hundred for redelivery, and because a hundred messages in one worker's hands is a hundred messages no other worker can process. The knob has different names (RabbitMQ's basic.qos prefetch count, Kafka's max.poll.records; SQS caps a single receive at ten). Keep it near the worker's actual concurrency.

Batching. The opposite pressure: sending, receiving, and acking messages in batches is often five to ten times cheaper and faster than one at a time, and producer clients can wait a few milliseconds to fill a batch before sending (Kafka's linger.ms). The trade is latency for throughput, and the sharp edge is partial failure: if message 7 of a batch of 10 fails, your code has to know which nine to ack.

Lag, not depth, as the metric. How many messages are waiting is less useful than how old the oldest one is. Kafka's consumer lag (the offset gap divided by the consume rate, which turns it into seconds) and SQS's ApproximateAgeOfOldestMessage are the numbers to graph and alert on, and when you autoscale workers, scale on backlog per worker or on message age. Scaling on raw depth oscillates: depth spikes, you add workers, depth crashes, you remove them, depth spikes again.

Priorities. "Send the password-reset email before the newsletter" wants a priority mechanism, and the simplest one is separate queues per priority with workers that drain the urgent one first (SQS has no priorities; RabbitMQ's classic queues take an x-max-priority argument and its quorum queues have built-in levels). Strict priority has a failure mode: under sustained load the low-priority lane starves entirely. Either weight the draining or give the low lane its own small worker pool.

Big payloads. Brokers cap message size (SQS at 1 MiB as of 2025, Kafka around 1 MB by default), and even under the cap, a queue full of 900 KB messages is slow and expensive. The claim-check pattern stores the payload in object storage and sends only a reference, and it's what the 2 GB TIFF should have been from the start: the worker downloads what it needs, and a corrupted file fails one download instead of one process.

Noisy tenants. One customer's bulk import of ten million records shouldn't delay everyone else's password resets. Options, from cheapest to most robust: a per-tenant rate limit on the producer (post #8), a queue per tenant or per tenant tier, or shuffle-sharding tenants across a set of queues so no two large tenants share all the same ones. SQS's fair queues are the hosted version of the same idea.

Visibility timeout versus long processing (Section 4): if legitimate work can take ten minutes, the timeout must exceed ten minutes, or the worker extends it with a heartbeat while it works, or the message gets processed twice concurrently and your idempotent consumer earns its keep.

Partition by key whenever order matters per entity (orders, users, accounts) and accept unordered everywhere else, because ordering costs throughput. Set retry caps low, DLQs on everything that matters, prefetch small, batches deliberate. And monitor the DLQ like a pager: a growing DLQ is either poison or a bug, and both want a human.

Section 7 — Later, precisely: delays, schedules, and retries with backoff

Not all "later" is "as soon as a worker is free." Sometimes later means precisely later:

Delayed jobs. "Retry this webhook in ten minutes." "Send the trial-ending reminder in three days." The message sits invisibly until its time comes. The delay is part of the message, not a worker's sleep call, because a sleeping worker is a worker that can't do anything else, and a sleep that's mid-way when the deploy rolls is a lost job. What the brokers actually offer is more limited than people assume: SQS delay queues and message timers max out at fifteen minutes, and FIFO queues don't support per-message timers at all; RabbitMQ's delayed-message exchange plugin stores its schedule on a single node (a node failure loses the delayed messages), isn't built for hundreds of thousands of pending messages or multi-day delays, and, having always been a separately shipped community plugin, was deprecated and archived in 2026, with its replacement available only in the commercial edition. For anything beyond minutes, use something designed for it: a scheduler service (EventBridge Scheduler on AWS), a scheduled_jobs table with a run_at column that the relay polls, or a workflow engine's durable timers (Section 9).

Scheduled jobs. "Every night at 2 AM, generate the report." "Every five minutes, check for stuck orders." This is cron's territory, but cron on one box is a single point of failure, and cron on ten boxes runs the job ten times. Production systems use a distributed scheduler with exactly one active runner, and "exactly one" is a leader election problem: the runners compete for a lock and only the holder runs the schedule. The lock can be a database advisory lock, a row with an expiry, a lease in etcd or ZooKeeper, or a Redis key with a TTL, and every one of them has the same sharp edge: the holder can pause (a long garbage collection, a stalled VM) past its lease's expiry, a second runner takes over, and now two runners believe they're the leader. The idempotency post (#6) has the fencing-token argument for surviving that. The practical rule that follows: a scheduled job will occasionally fire twice, so the job itself must be idempotent, and the job should usually just enqueue the real work rather than doing it. The schedule is the trigger; the queue is the muscle.

That's the retry discipline in one picture, and it's the twin of the resilience post's backoff chapter: exponential backoff with jitter, on every retry, no exceptions. The version that works best, from AWS's analysis, is full jitter: each wait is a random duration between zero and min(cap, base × 2^attempt), so the ceiling doubles each time but the actual wait is spread across the whole range. The jitter isn't a sprinkle on top of a fixed delay; it's the whole delay, because the point is that a thousand workers never retry in the same instant. A payment service that skips this goes down for four minutes, every client retries immediately in a tight loop, and when the service comes back it's greeted by a stampede many times larger than normal traffic. It falls over again. Then again. The outage lasts forty minutes, of which thirty-six are self-inflicted. With backoff and jitter, the retries trickle in over minutes and the service recovers on the first try. The retry that kills a recovering service is the immediate one.

Cap the attempts (three to five is the usual range) and then the message goes to the DLQ (Section 6), not into infinite retry. Infinite retry is a slow-motion outage: a broken downstream means every message retries forever, workers churn, and the queue never drains. A retry policy without a cap is a hope, not a policy. Give retries a budget as well as a cap: no more than some fraction of total traffic may be retries at any moment, so a sick downstream isn't buried under retries of its own failures. And remember that retries stack across layers: a consumer that makes three attempts, calling a service whose client retries three times, calling a database whose driver retries three times, turns one message into up to 27 attempts. The resilience post has the arithmetic. Make retries visible: log the attempt number, alert on retry-rate spikes, because a sudden jump in retries is often the first signal that a downstream is sick, minutes before its own alarms fire.

The pattern is the same whether it's a queue consumer, an HTTP client, or a saga step: wait longer each time, randomize, stop eventually.

Section 8 — Backpressure: when producers outrun consumers

Every queue is a bet: that on average, consumption keeps up with production. Averages lie. The marketing campaign multiplies signups by ten for a weekend. A downstream slows every worker by 30%. A deploy halves the fleet for an hour. And then the arithmetic is merciless: 1,100 messages arriving per second, 1,000 processed per second, the queue grows by 100 a second, 360,000 an hour. The queue is doing its job, holding messages, right up until the holding becomes the emergency.

First, the word, because it gets used for everything. Backpressure is a signal that flows against the direction of data: the consumer telling the producer "slow down," and the producer actually slowing. TCP does it with its receive window; reactive-streams libraries do it with request(n); Kafka's producer does it by blocking when its local buffer fills. A queue with an unbounded buffer has no backpressure at all; it just absorbs until it can't. So the three options below are what you do when the buffer is filling, and only one of them is backpressure in the strict sense. You choose among them in a design review, not during the incident:

1. Buffer. Add consumers, drain faster. Works when the surge is temporary and the downstream can take it; autoscaling workers on lag (Section 6) is the automated version. But buffering has a limit: if the arrival rate permanently exceeds capacity, you're renting a bigger pile.
2. Drop. Deliberately discard low-value work. Analytics events can be sampled; the tenth "user viewed page" event this minute can go. Dropping is a business decision disguised as an engineering one. It must be chosen per queue, in advance, with the product's blessing, and every drop must be counted, because "we dropped 40% of analytics on Tuesday" is a fact someone needs.
3. Push back. Slow the producers: bounded buffers that block, or, at the API edge, HTTP 429 with a Retry-After header (the rate limiting post's, #8, territory). This is the truthful option: the system admits it's full instead of accepting work it can't do. For user-facing producers, pushing back degrades gracefully. For internal ones, it moves the queue's problem to the caller, which is correct, because the caller can decide what matters.

The way this goes wrong slowly: a notification service that's 2% underwater, producers adding 2% more messages a day than the workers drain, a gap invisible on any single day's dashboard. On its own that gap would grow the backlog by about half an hour a day, three or four hours by the end of the week, which someone might catch. What actually happens is that it compounds. As lag grows, the downstream email provider starts timing out under the steadier load, timeouts become retries, retries widen the gap, and by Friday the "your order shipped" email arrives at midnight and the "flash sale ends tonight" email arrives Saturday. The postmortem finds the underlying gap has existed for months. The fix is an alert on the growth rate of lag, not just on lag: a flat-but-high queue is a spike; a steadily growing one is a capacity emergency with a date.

Lag is the vital sign. Not worker CPU, not message rate: the age of the oldest message, or depth divided by consume rate, which is the same number in seconds (360,000 messages at 1,000 a second is six minutes of lag, the answer to "how stale is the newest work?"). Alert on lag crossing the work's freshness budget and on lag growing over time. And size consumer capacity for a lag target rather than for the average arrival rate: decide how stale the work may get (six minutes is fine for analytics, fatal for fraud checks), then provision enough consumers that the peak-hour backlog drains inside that budget, with the queue absorbing the difference.

Document the policy per queue before the spike: buffer, drop, or push back, and for drop, what may be dropped and who approved it. Set the lag alerts on day one. The queue's promise is that it turns capacity emergencies into latency bills. Just remember: an unbounded latency bill is an emergency with slower paperwork.

Section 9 — Going deep: the principal-level toolkit

Everything so far was machinery. This section is judgment: the calls that separate a system that works from one that survives its own success. How to coordinate multi-step workflows, how to read the lag arithmetic, which vendor claims to believe, the traps in event-driven design, what async does to the user and to the person debugging it, and when not to go async at all.

Choreography versus orchestration. A multi-step workflow (order placed, payment charged, inventory reserved, warehouse notified, email sent) can be coordinated two ways. In choreography, there's no boss: each service listens for events and reacts. The order service emits order.placed; the payment service hears it and charges; it emits payment.charged; the warehouse hears that and ships. Nobody owns the flow. In orchestration, a central orchestrator runs the show: it calls step 1, waits, calls step 2, and when step 3 fails it runs the compensating actions for steps 2 and 1. That's the saga pattern from Section 5, and workflow engines (Temporal and its ancestor Cadence, AWS Step Functions, the Netflix-born Conductor OSS) are orchestration as a product: they give you durable timers ("wait three days, then continue"), automatic retries per step, and a complete history that answers "what state is order 4821 in, and what happened at step three?" What they cost is a stateful service you have to run or pay for, and a programming model your team has to learn.

Choreography is beautifully decoupled (add a new reaction, the fraud service starts listening, without touching anything) and beautifully illegible: the workflow exists nowhere. It's spread across five codebases' event handlers. Debugging is archaeology ("who emitted what, when, and who heard it?"), and onboarding a new engineer onto a choreographed flow takes weeks. Orchestration inverts the trade: the flow is written down in one place, failures and compensations are explicit, but the orchestrator is a component to build, scale, and keep highly available, and every step's latency now includes a round trip to the boss. The rule of thumb: choreography for simple flows (three steps or fewer, no rollback needed); orchestration when the flow has branches, compensations, timers, or anyone will ever ask what state a particular order is in. Whichever you pick, the idempotency post's rule follows you: every step and every compensation must be idempotent, because steps retry.

The lag arithmetic, and the law behind it. Section 8 gave you the vital sign; here's why it works. If messages arrive at λ per second and each spends W seconds in the system (waiting plus processing), then on average there are L = λ × W messages in the system. That's Little's law, and it's the estimation post's (#13) favorite equation. Rearranged, W = L ÷ λ: lag equals depth divided by throughput, which is the formula from Section 8. And the growth rule is simpler still, plain conservation: the pile grows by (arrivals − departures) per second, so a queue that's growing has an arrival rate above its consume rate, and no amount of tuning changes that until departures beat arrivals. 360,000 messages at 1,000 per second is 360 seconds of lag; the newest message waits six minutes. That's the number for the dashboard, because "depth: 360,000" means nothing to a human and "lag: 6 minutes" means everything.

Exactly-once versus effectively-once, one last time. You'll keep meeting vendors who say "exactly-once." Kafka's transactions give you exactly-once processing for pipelines whose input and output are both Kafka topics, with an idempotent producer and a read_committed consumer. SQS FIFO deduplicates sends within a five-minute window. Both are real, scoped, useful claims, and neither changes your architecture, because the boundary between the broker and the code that sends the email or charges the card is still at-least-once, and your consumer still dedups (Section 4). Design for effectively-once at the edges regardless of what the pipe promises. The pipe's guarantee is a bonus, not a foundation.

Event-driven pitfalls: the three that bite. First, ordering: a log gives you order within a partition, and nothing across partitions. If your design needs "every consumer sees events in global order," redesign; global order at scale costs you all your throughput, and Section 6's per-key ordering is the version that works. Second, replay: the log keeps history, so a new consumer can rewind to day one, which is a superpower for backfills and a loaded gun in two ways. For privacy, "delete my data" against a log is either retention expiry (the data ages out), compaction with a tombstone (for keyed, compacted topics), or crypto-shredding (encrypt each user's events with a per-user key and destroy the key), and you need to know which one your topics support before you promise anyone a deletion. For reprocessing, replaying six months of events through a fixed consumer is a migration, and it re-runs every side effect unless you plan for it: a replay mode flag that suppresses emails and external calls, a shadow consumer that computes results without acting, idempotency keys that span the replay, and a DLQ redrive that doesn't resurrect deleted work. Third, schema evolution: the producer ships event v2 with a renamed field, and the v1 consumer, deployed last quarter and still running, chokes. Events are a public API. The tooling for that is a schema registry (Confluent's, or the AWS Glue one) holding Avro, Protobuf, or JSON Schema definitions, with a compatibility rule enforced on every new version: backward compatible (new consumers can read old events), forward compatible (old consumers can read new events), or full. The operational rule that follows is to pick a compatibility mode and deploy in its order (with the usual backward mode, consumers can read old events, so consumers deploy first; with forward, producers deploy first), and the design rule is additive changes only: new optional fields, never a renamed or repurposed one. The Knight Capital lesson from the idempotency post applies double here: a flag whose meaning changed cost them more than $440 million in forty-five minutes.

What async does to the user. "Later" is a promise the interface has to keep visibly. The HTTP shape is 202 Accepted plus a status resource (GET /jobs/4821) the client can poll, or a webhook or push notification when the work completes. In the interface it's optimistic UI (show the order as placed, mark the email as "sending") with a real pending state rather than a fake completed one. And it collides with the consistency the user expects: "I just placed an order, where's my order?" now races the queue. The replication post (#5) calls this read-your-writes, and the answers are the same here: route the user's next read to the source that has their write, or show the pending state honestly, or write the user-visible record synchronously and defer only the invisible work. What you don't do is let the user refresh into an empty page and wonder.

Tracing across the queue. A request's work is now scattered across time and machines: the charge at 10:00, the email at 10:04, the warehouse event at 10:06, on three hosts. Without distributed tracing, "why didn't the email send?" is a multi-system scavenger hunt. The mechanism is simple and constantly forgotten: the producer writes the trace context (the W3C traceparent value) into the message's headers or attributes, and the consumer reads it back and starts its span as a child of the producer's. For batch consumers, OpenTelemetry's span links let one consumer span point at the many producer spans it's handling. Put the message ID in every log line the consumer writes. The observability post (#9) covers the rest of the machinery; this is the one habit that keeps traces from going cold at the queue.

Security and testing, briefly, because they're skipped. A queue is an attack surface: control who may publish and who may consume (IAM policies, Kafka ACLs), encrypt in transit and at rest, keep personal data out of messages where you can (the claim-check pattern helps here too), and treat message contents as untrusted input in the consumer, exactly as the security post (#12) says to treat any input. For testing, run the real broker locally (LocalStack, testcontainers), write contract tests against the schema registry, and inject the failures on purpose: duplicate every message in a test environment, deliver some out of order, plant a poison message, and check that the inbox, the ordering keys, and the DLQ behave. Async assertions need a wait-and-poll helper; a test that checks the side effect immediately after enqueueing is testing the race.

When not to go async. Async has costs, and a principal names them. The debugging cost above. The reasoning cost: "did it happen yet?" becomes a real user question, and support has to be able to answer it. The consistency cost, from the user-experience paragraph. So don't go async when the user needs the answer now (Section 2's question, still the law), when the flow is two steps and always fast (a queue for a 50 ms job is ceremony), or when the work's value expires in seconds (real-time bidding, live collaboration).

Async is a loan against future debugging. Take it when the interest is worth it.

The team that choreographs everything, twelve services reacting to events, no orchestrator, no tracing, has an elegant year. Then a payment succeeds, the warehouse never ships, and the investigation takes three engineers two days: the event was emitted, consumed, and dropped by a handler with a swallowed exception, and nothing anywhere recorded the flow's state. They introduce an orchestrator for the money path, not for all paths, just the one where "what state is order 4821 in?" is a question with dollar signs, and keep choreography for the rest. The principal move is not to pick a side but to draw the line between the flows that need a boss and the ones that don't.

And the gate, stated the way the idempotency post stated its own: a new queue ships if and only if it has a DLQ (or a documented reason not to), a lag alert, a documented drop-or-push-back policy, an idempotent consumer, and trace context in its messages. "We'll add monitoring later" is not a story. Five minutes in the design review, or a page in the middle of the night: those are the options, and they're priced accordingly.

Do It Later, On Purpose, distilled

For the skimmers and the revisitors: everything above, on one page.

Key numbers and rules:

Every trade-off, in one table:

Three ideas to take with you:

1. The only question is "does the user need the result now?" Everything else (the queue, the workers, the DLQ, the backoff) is machinery in service of that one decision. Most slow endpoints aren't slow because any step is slow; they're slow because work nobody's waiting for is spending the waiter's time.

2. At-least-once delivery plus an idempotent consumer equals effectively-once, and the dedup is always at the edge. The pipe's guarantees are bonuses, not foundations. If you can't point to the inbox check in your consumer, and to the transaction it shares with the effect, you don't have reliability, you have optimism.

3. A queue without a DLQ, a lag alert, a drop-or-push-back policy, an idempotent consumer, and trace context is a hope, not a design. The boring parts are the design. Set them in the review, when it's cheap.

Further reading

• Amazon SQS: Developer Guide. Standard vs FIFO queues, visibility timeouts, dead-letter queues, message timers, and the quotas quoted in this post.
• Apache Kafka: documentation. The log-based shape: partitions, consumer groups, offsets, retention, and the delivery-semantics section behind Section 4.
• Confluent: Message delivery guarantees. What Kafka's "exactly-once" actually covers.
• RabbitMQ: Consumers and Dead letter exchanges. Acknowledgement, prefetch, the acknowledgement timeout, and dead-lettering as RabbitMQ does them.
• Debezium: Reliable microservices data exchange with the outbox pattern. The outbox with change data capture; behind Section 5.
• Marc Brooker: Exponential Backoff and Jitter (AWS Architecture Blog). The analysis behind full jitter in Section 7.
• Amazon Builders' Library: Timeouts, retries, and backoff with jitter. The retry side of the contract; the resilience post's companion.
• Amazon Builders' Library: Using load shedding to avoid overload. Shedding as a designed behavior, not a panic; behind Section 8.
• Temporal: Activities. What a workflow engine gives you (durable timers, retries, history) and why activities must be idempotent; behind Section 9.
• Designing Data-Intensive Applications: Martin Kleppmann. Logs, exactly-once versus effectively-once, and stream processing; the deep end of Sections 3, 4, and 9. A second edition (with Chris Riccomini) is out.

Where you'll meet this

This post leans on the idempotency post (#6) twice: its Section 4 walked through why exactly-once delivery is impossible, which is why every consumer here dedups, and its inbox and outbox sections are this post's Sections 4 and 5, the same pattern from both sides. It's the resilience post's (#2) bulkhead made concrete: the queue is the wall between "the email service is down" and "checkout is down," and Section 7's backoff is that post's retry chapter with the queue as the stage. The sharding post's (#3) sagas reappeared in Section 9 as orchestration, and its partition-by-key idea as Section 6's ordering. The rate limiting post (#8) is where "push back" lives once it reaches the API edge, the replication post (#5) owns the read-your-writes problem async creates, and the observability post (#9) is where the trace context in Section 9 ends up. And the URL shortener design was async from the start: the interview's answer to "how do click analytics avoid slowing the redirect?" was this post's one question. The user needs the redirect now; the analytics can wait. Next up is load balancing (#11), the box in every diagram that decides which worker gets the message in the first place.

Keep exploring: the Core Concepts series

Every post in the series stands alone. Read them in any order.

• From Laptop to Billions of Clicks: Designing a URL Shortener in 11 Steps — the anchor: one design, every concept under load.
• #1 The 100:1 Superpower: Caching — the fastest request is the one you never make.
• #2 The Blast Radius: Surviving the Day Your Dependencies Fail — staying up when everything you depend on goes down.
• #3 Divide and Conquer: Sharding and Partitioning — splitting one database into many without losing your mind.
• #4 The Snowflake Problem: Unique IDs at Scale — naming things when millions are born every second.
• #5 Copies of the Truth: Replication — keeping copies of your data that actually agree.
• #6 Do No Harm Twice: Idempotency — making "just retry it" safe.
• #7 The Copy at the Doorstep: CDNs and Edge Computing — serving from next door instead of across the ocean.
• #8 The Bouncer's Math: Rate Limiting — saying no politely, at scale.
• #9 What Broke at 3 AM: Observability, p99, and Useful Alerts — knowing what's wrong before your users tell you.
• #10 Do It Later, On Purpose: Async Processing and Queues — the work the user doesn't have to wait for. (this post)
• #11 The Traffic Cop: Load Balancing — the box in every diagram nobody explains.
• #12 Assume They're Already Knocking: Security and Abuse at Scale — designing for the users who are designing against you.
• #13 Do the Math First: Estimation for System Design — the two minutes of arithmetic that choose the architecture.
• #14 The Whiteboard Playbook: Taking On Any System Design Challenge — the whole series in forty-five minutes.

Blueprints of Scale — Core Concepts #10. If this helped, the best thanks is a share with someone who's learning.

This post is that method, written down. It's the capstone of the Core Concepts series because every other post in the series is a tool this playbook reaches for. System design is a procedure rather than a talent, and procedures can be learned.

Here's the route: why methodology beats brilliance at the whiteboard, and what interviewers are actually scoring; clarifying requirements and controlling scope before drawing a box, including what to do when the interviewer pushes back; doing the math second, always; the skeleton nearly every system shares, plus the four sketches that go on it (the API, the data model, the storage choice, and the data flow); how to pick the two or three deep dives a problem deserves, with the toolbox of all thirteen concept posts mapped to where each plugs in, and what to do when you're running long or don't know a technology; how the method bends across a dozen kinds of systems; breaking your own design on purpose; the principal close (10× thinking, trade-off statements, cost, migrations, consistency per operation); how junior, senior, staff, and principal answers to the same problem differ; and how to practice.

If you've never done one of these, start at Section 1; the first two sections assume nothing. Sections 3 through 8 are the machinery working engineers use under pressure. Section 9 is level calibration. The post refers back to the URL shortener interview throughout, because that interview is this methodology in action, but it doesn't require having read it. The one-page checklist you can carry into a real interview is at the end, under The Whiteboard Playbook, distilled, and every diagram is also described in the text around it.

Section 1 — The blank whiteboard

Two candidates, same prompt, same whiteboard, same forty-five minutes. "Design a URL shortener." Candidate A uncaps the marker and starts drawing: a box labeled "server," a cylinder labeled "DB," arrows everywhere. Ten minutes in there's a handsome diagram and zero decisions. The interviewer asks "how many requests per second?" and A says "a lot." The diagram gets more boxes. At minute forty there's a beautiful picture of nothing in particular.

Candidate B doesn't draw a single box for the first four minutes. She asks: shorten and redirect, or also analytics? Read-to-write ratio? Latency target? Then she does the math out loud (40 writes a second, 4,000 reads) and says "so the reads are the whole problem, and they're cacheable." Her diagram has half the boxes of A's. The interviewer writes "strong hire" on the feedback form.

Same talent pool. Same prompt. The difference wasn't brilliance; it was procedure. Candidate A designed by drawing. Candidate B designed by deciding, in an order that compounds: questions first, numbers second, boxes third, and the hard parts exactly where the numbers point.

This is the whole playbook in six moves, and it's the shape of every section that follows. Notice what it doesn't start with: boxes. The marker is the last tool you pick up, not the first. Drawing feels like progress and is usually procrastination. A diagram of an undesigned system is confusion with better geometry.

A whiteboard interview is not a drawing test but a decision test with a drawing component.

It helps to know what's on the other side of the table. In my experience most rubrics score roughly the same six things: whether you clarified requirements and scoped the problem; whether the high-level design is sound; whether you went deep somewhere and got it right; whether you justified trade-offs rather than just naming choices; how you communicated (structure, legibility, taking hints); and, past the senior level, whether the thing could be operated and scaled. "Strong hire" in the interviewer's notes usually means something specific: I never had to steer. The candidate found the hard part unaided, the trade-offs came out unprompted, and the failure modes got discussed without being asked for. "Hire" means they got there with hints. And the whole thing is calibrated to the level you're interviewing for: the same answer can be a strong hire for a mid-level role and a no-hire for a staff role, which Section 9 is about.

The interviewer is simulating working with you. Every question you ask is them thinking "this is what a design review with this person would feel like." Every number you compute is "this person won't build the wrong thing for six months." The artifact is the diagram. The product is your judgment, narrated.

Section 2 — Clarify before you draw

The first five minutes decide the next forty. The usual way to lose them is to build an elegant system for the wrong product: twenty minutes into "design WhatsApp," a candidate has a beautiful text-messaging design, and the interviewer says "what about voice calls?" Nobody asked. The system was good. It was good for the wrong product.

Requirements come in two flavors, and you need both before the marker moves:

Functional: what the system does. Shorten a URL, redirect a click. Send a message, deliver it. Upload a video, play it back. List these out, then negotiate scope down. "Custom aliases and link expiry are nice-to-haves; can we park them?" Interviewers like this. It shows you know that every feature is boxes on the diagram and weeks on the roadmap, and that you're spending the budget deliberately. Saying "out of scope" explicitly is a senior signal. It proves you're designing the system that was asked for, not the one you wish had been.

Two scoping traps worth naming. "Design Twitter" is not a system; it's a company. Propose the cut yourself: "I'll treat this as the home timeline: post, follow, read the feed, and I'll leave search, DMs, and ads out unless you want them." Conversely, when you're handed a feature of a system ("design the typeahead for search"), assume the surrounding platform exists and name the interfaces you depend on, rather than redesigning the whole product around it.

Non-functional: what the system promises. This is where interviews are actually graded, because candidates skip it:

(The p50 and p99 are the median and the 99th-percentile latency: half of requests finish within the p50, 99% within the p99. The p99 is where problems show up first, and the observability post, #9, is about why.)

Ask five or six of these, no more, and state the rest as assumptions. You're scoping, not stalling. And write the answers on the board, because verbal requirements evaporate and written ones become constraints you can point at when you make a trade-off in minute thirty: "we said p99 under 100 ms, so the cross-region write has to be async."

The trap on both sides: the candidate who asks nothing builds the wrong system confidently, and the candidate who asks everything burns fifteen minutes on "what's the exact retention policy for deleted accounts?", a question with no design consequences. Ask the questions whose answers change boxes. If the answer wouldn't move a single arrow, skip it.

Watch it work on a fresh prompt, "design a notification service," in under two minutes:

Me: "What kinds of notifications: push, email, SMS, all three?"

Them: "Push and email to start."

Me: "Volume? And is delivery time-sensitive? A fraud alert can't wait an hour; a weekly digest can."

Them: "About 50 million a day. Fraud alerts within a minute; digests whenever."

Me: "Is the volume smooth, or does it arrive in campaigns? Ten million digests released at nine in the morning is a different problem from ten million spread across the day."

Them: "Campaigns, mostly."

Me: "So two priority lanes rather than one queue, sized for the campaign burst rather than the daily average, and SMS is out of scope for now?"

Them: "Correct."

Five questions, and the design already has its shape: priority lanes (that's the async processing post, #10, knocking), a burst-shaped capacity target rather than the misleading average (50 million a day is about 600 a second on average, and that number would have led you to size a system that falls over at nine every morning), and an explicit scope cut. The clarification isn't small talk. It's the first design decisions, disguised as questions.

When the interviewer pushes back. Three things happen in this phase that candidates misread. "Assume X" or "let's not go there" is a gift: park it in one sentence and move on. "Are you sure about that?" is a probe, not a verdict; restate your reasoning, and then actually reconsider, because sometimes they're steering you toward a hole and sometimes they're checking whether you fold. And somewhere around minute fifteen, ask: "is there an area you'd like me to spend the time on?" It costs nothing, it often hands you the deep dive they had planned, and it's exactly what you'd do with a colleague.

Section 3 — Do the math

The interview's Step 2 was the least glamorous and most decisive. "100 million new URLs a month" became 40 writes a second; "100:1" made it 4,000 reads a second; 500 bytes a record over five years landed in the low terabytes. Three calculations, and the architecture half-designed itself: writes are boring, reads are everything, storage fits on a handful of machines. Do the math first and the architecture half-designs itself. Skip it and you'll optimize the wrong path for forty minutes.

The estimation post (#13) is the full toolkit. Here's the compressed version you run at the whiteboard:

Four numbers carry most interviews. Requests per second: divide the daily total by 86,400 (call it 10⁵ for mental math, knowing it understates by about 14%), then multiply by two or three for peak, or more if the traffic arrives in campaigns. Storage: record size × count × years × replication factor; forgetting the replication factor is the classic faceplant. Bandwidth: response bytes × rps, the number that kills "we'll just serve video from the app server." And the ratio: read-to-write decides caching versus write-optimization more than any other single fact, though the ratio alone doesn't decide whether you need a cache at all. 100:1 at ten requests a second needs nothing.

Two habits separate people who do math from people who perform math. First, round aggressively: one significant figure, always. "100M a month is about 40 a second" is more useful than "38.58," because the point is the order of magnitude, not the digit. Second, sanity-check against an anchor: "4,000 reads a second is one Redis node's worth of throughput, plus a replica so it isn't a single point of failure, and the hot set has to fit in its memory." Anchors turn abstract numbers into design decisions. A small anchor sheet is worth memorizing, and the estimation post has the full version: one Postgres primary does thousands of transactions a second and tens of thousands of indexed reads with the hot set in memory; one Redis node does 100,000-plus reads a second; a 10 Gbps link is 1.25 GB/s; a cross-region round trip is 30 to 150 ms; egress costs roughly $0.05 to $0.12 a gigabyte, depending on the cloud and the volume tier.

The failure mode has a name in every interview debrief: "no numbers." The candidate draws a beautiful multi-region architecture for a system doing 200 requests a second. Everything is correct and everything is waste, and they'd have known if they'd divided by 86,400 first. Numbers are how you avoid designing a spaceship to cross the street.

Which raises the question of how big to design in the first place, because the advice "design for 10×" and the advice "start with a monolith" both float around, and they seem to contradict. They don't, because they're about different axes. Martin Fowler's Monolith First is about service boundaries: don't split into microservices before you understand the domain. The 10× advice is about capacity: know where the design stops working. The reconciliation is the move Candidate B made: size to the numbers you computed, not to Google's. "At 4,000 rps this is one Postgres with a replica and one Redis node; the seam where I'd shard later is the short-key hash, and here's what I'd have to change." Naming what you deliberately don't build yet, and where the seam is, is one of the most senior sentences in the interview.

Section 4 — The skeleton

Most systems share the same skeleton: client, load balancer, stateless API servers, a cache, a database, background workers, and a queue between them. The URL shortener, a food-delivery backend, a notification service: different labels on the boxes, same bones. Starting from the skeleton isn't unoriginal; it's truthful. Originality at the whiteboard is usually an unfamiliar way to be wrong.

Draw this in minute ten and you've bought yourself something precious: a complete system with no decisions in it yet. Every arrow is now a question you get to answer deliberately, and the deep dives in the next section are you picking which arrows deserve the attention. Note the dotted box. Observability isn't a deep dive you might get to; it's part of the skeleton. Every service emits request rate, error rate, and latency (the "RED" metrics from the observability post), every request carries a trace ID that survives the queue, and the latency and availability targets you wrote on the board in minute three become the service-level objectives you'll alert on. Drawing that box early closes a loop the rest of the interview keeps promising.

Four things to sketch onto the skeleton before going deep:

The API, as contracts. Contracts, not code. POST /shorten {url} → {code}, GET /{code} → 302. Five lines, and suddenly the design is discussable. Three refinements turn a sketch into a senior sketch. First, put an idempotency key on every mutating endpoint (POST /shorten {url, idempotency_key}), because "what does the client send on retry?" is where the idempotency post (#6) walks in, and it belongs in the API sketch, not discovered in minute forty. Second, for anything that returns a list, say "cursor pagination," because offset pagination breaks the moment the list changes underneath the reader and every interviewer knows it. Third, name the error contract: which errors are retryable (503, and 429 with a Retry-After header saying how long to wait), which aren't (400, 404, 409 for a conflicting alias), and how the API is versioned (a path prefix or a header) so it can change without breaking clients. An API sketch is cheap and it forces every ambiguity into the open while it's still erasable.

The data model, as three entities. Draw the two or three tables that matter, their primary keys, and the access pattern each index serves. For the shortener: urls(short_key PK, long_url, owner_id, created_at, expires_at), and note that "list my links" needs a secondary index on owner_id that doesn't live with the shard key, which is the kind of thing you want to discover now rather than in the sharding discussion. Say which column is the shard key and which one denormalization you're accepting. Ten seconds of schema surfaces hot partitions, secondary-index problems, and the join you can't do, before anyone draws a database cylinder.

The storage decision, framed rather than solved. Storage engines are a deep subject, and the whiteboard needs the decision frame, not the dissertation:

Pick from the access pattern, say why in one sentence, and move on. (The two branches the text hasn't mentioned: a document store when the schema is flexible and each record is read whole, and a time-series store when data is append-heavy and time-ordered and you'll want downsampling and retention rules.) "Key lookups by short code, no joins: key-value store, one node with replicas now, sharded by key when it outgrows one box." A few rules ride along with the frame and are worth saying because they're what separates "I picked Postgres" from "I know what Postgres costs": every index you add speeds a read and taxes every write, so name the two indexes you'd create; a search index is a second system fed from the primary by change-data-capture (CDC, streaming the database's change log) or an outbox, never bolted onto the primary with LIKE '%query%'; analytics never run on the primary (that's OLTP versus OLAP: transactional versus analytical workloads, and the second gets its own columnar store fed the same way); large blobs live in object storage with only their metadata in the database; and read replicas come before sharding, because they're a configuration change and sharding is a migration. A principal doesn't recite B-tree internals at the whiteboard. A principal picks correctly in ten seconds and spends the saved thirty minutes on the actual hard problem. If the interviewer wants engine depth, they'll ask, and that's them choosing your deep dive for you.

The data flow, traced once. Pick the most important request (the redirect, the message send, the checkout) and walk it through every box. This is where missing pieces surface: "wait, who generates the short code?" (the Snowflake Problem, #4), "where do click events go?" (the async processing post, #10). Tracing one request end to end catches more design holes than staring at the diagram for ten minutes.

And the API sketch pays for itself immediately. Take the shortener's write path: POST /shorten {url, idempotency_key} → {code}. That one field forces three decisions early: the client generates the key per intent rather than per attempt, the server stores key→response with a time-to-live, and the retry path returns the stored response instead of minting a second code. None of that is visible in a box labeled "API." The boxes show structure. The sketches show behavior.

Section 5 — Choose your battles

You cannot go deep everywhere in forty-five minutes. Nobody can. The candidate who tries covers seven topics at surface level and demonstrates nothing. The candidate who picks two deep dives and nails them demonstrates judgment, which is the actual skill being hired for. The interview knew this: key generation got two full steps (4 and 6), and sharding in Step 7 got the reasoning it needed and no more.

So how do you choose? Two rules, in order.

Follow the ratio. The estimation math tells you where the pain lives. Read-heavy at scale: the deep dive is caching, hot keys, and the read path. Write-heavy: partitioning, backpressure, and the write path. The numbers point at the bottleneck, and the bottleneck is the deep dive. This is why the math comes before the boxes.

Follow the novelty. Of the remaining candidates, go deep where this problem is unusual. Every system needs a database; not every problem's database is interesting. The URL shortener's database was boring (key lookups, shard by key, done); its key generation was the puzzle, so that's where the minutes went. Depth belongs on the parts a generic skeleton doesn't solve.

And now the payoff of the whole series, the toolbox. Thirteen concept posts, each mapped to the moment in the method where you reach for it:

The router's two other branches, coordination (IDs and idempotency) and distance (edge and regions), are where the pain lives when the problem is about agreement between machines or about geography rather than about read or write volume. Name the tools you're not going deep on, briefly ("replication: async cross-region, standard; moving on"), because naming them proves you saw the whole board. Breadth is shown by naming; depth is shown by choosing. The interviewer needs both, and this is how you give both in forty-five minutes.

Two situations that come up in this phase and that nobody prepares for:

You're running long. Announce checkpoints as you go ("that's the skeleton; I'm going to spend the next fifteen minutes on key generation and the read path"). If the deep dive hasn't started by minute twenty, cut the skeleton short and say what you skipped. Never drop the failure pass entirely; a three-minute version beats none. And when it's tight, ask: "ten minutes left. Would you rather I go deeper on the cache or walk the failure modes?" The interviewer would rather choose than watch you guess.

You don't know a technology they mention. Say so, and reason from properties. "I haven't run Cassandra in production, so I'll reason from what a log-structured wide-column store gives you: fast writes, partition-key lookups, no joins, tunable consistency." Interviewers grade reasoning far above name-dropping. Bluffing is the fastest way to fail, because the follow-up question always finds it.

Section 6 — Every type of system

The method doesn't change. The emphasis does. A chat app and a payment ledger go through the same six moves, but the moves land on different boxes. Here's the field guide: the common system types, what each forces you to emphasize, and the trap that catches people in each.

Several of these deserve a paragraph, because the traps are where interviews are lost:

Write-heavy flips the shortener's instincts. The interview taught "reads are everything"; for metrics ingestion, writes are everything, and every instinct reverses. You batch, you buffer, you accept eventual everything, you pick storage by write throughput (log-structured engines, the LSM trees under Cassandra and RocksDB, which turn random writes into sequential ones), and backpressure (the mechanism by which a slow consumer makes the producer slow down instead of drowning it) isn't a failure mode, it's the design. The async processing post is the whole post for this type.

Consistency-critical is where you say the sentence out loud. "The ledger write is atomic and strongly consistent; the receipt email is eventually consistent; the balance the user sees on the dashboard can lag by a second." That per-operation consistency call, from the replication post (#5), is worth more than ten minutes of diagram. The nuance a principal adds: real payment systems are strongly consistent within the ledger and eventually consistent across services, held together by idempotent operations and a reconciliation job that catches drift. "Eventual consistency for money" isn't automatically wrong; "eventual consistency for the ledger with no reconciliation" is. Anyone who gives one consistency answer for the whole system hasn't thought hard enough.

Graph systems punish the generic skeleton. The skeleton's database box hides the real question: fan-out on write (precompute each follower's feed when a post is made; fast reads, and a celebrity's post becomes millions of timeline writes) versus fan-out on read (cheap writes, and every timeline load has to gather posts from everyone the reader follows, which gets slow for a celebrity's millions of followers all reading at once). The sharding post (#3) named the celebrity problem; here it becomes the deep dive, and the standard answer is a hybrid: fan-out on write for ordinary accounts, fan-out on read for the accounts with millions of followers. The trap isn't joins as such; indexed adjacency joins are fine at modest depth. The trap is unbounded traversal, the "friends of friends of friends" query with no limit, and the bigger trap is answering it by proposing a graph database when most social graphs at scale are sharded adjacency lists with a cache.

Real-time systems are won or lost on the latency budget. The method's estimation step becomes a budget: 100 ms end to end across five hops means about 20 ms each, and the accurate version of that arithmetic (the estimation post, #13, Section 7) is that per-hop p99s don't simply add: for a chain of hops the sum of p99s is a pessimistic ceiling, while for anything that fans out and waits for every leg you have to reserve more for the tail than the even split suggests. The deep dive is fan-out: one message to 10,000 subscribers can't be 10,000 sequential writes. The trap people name is polling, but long-polling is a legitimate design at small scale; the real traps are the unbounded queue that converts "real-time" into "eventual" under load, and forgetting that a persistent WebSocket connection makes your servers stateful, which drags in connection affinity at the load balancer (#11), reconnect storms on deploy, and the question of where subscription state lives when a server dies.

Media-heavy systems are a bytes problem. The bandwidth math dominates: 2 MB per image × 10,000 uploads a minute is about 330 MB/s sustained, around 2.7 Gbps, which a few servers with fast network cards could proxy. The reason you don't is that it's pointless: every byte handled twice, stateless compute scaled to shovel bytes it never looks at, and long-lived connections pinned on slow mobile uploads. So uploads go directly to object storage through a presigned URL the app server mints (the security post, #12, has the recipe), the app server gets a callback when the upload finishes, transcoding runs in async workers (#10), and serving happens from the CDN (#7). The app tier owns metadata, permissions, and orchestration; the bytes never pass through it. And the number that truly forces this design is the read side: each image viewed a hundred times is 33 GB/s, which is nothing but the edge.

Search is two systems, not one. There's the serving path (query → ranked results in milliseconds) and the indexing path (ingest, parse, build the inverted index, a mapping from each term to the documents containing it, as a pipeline that runs continuously or in batches). Candidates who draw one system build neither. The deep dive is the index: freshness versus index-build cost, and ranking as a separate concern from retrieval. The trap, LIKE against the primary, is the fastest way to tell the interviewer you've never built search.

Geo systems need a spatial index, and the ride-sharing example in Section 9 is one. "Drivers near me" against a table of raw coordinates is a full scan. Geohashes and S2 cells turn two-dimensional locations into sortable one-dimensional keys so that "nearby" becomes a prefix range query; quadtrees do the same job in memory. The other half is the write rate: every active driver reports a position every few seconds, so the location store is write-heavy and ephemeral (a location from five minutes ago is worthless), which points at an in-memory store with short expiry rather than a durable database.

A closing note for this section: don't memorize twelve architectures. Memorize a handful of questions. "Where's the ratio? What can't be stale? What's the fan-out? Where are the bytes? What's the spatial or temporal shape of the data?" The method generates the architecture; the type tells you which questions bite.

Section 7 — Break it on purpose

The design is done when you've tried to kill it. The interview's Step 11 was the highest-signal ten minutes on the board: the cache dies at peak, a region goes dark, hostile traffic arrives. Junior designs describe the happy path. Principal designs describe what happens when the happy path ends, and this pass is how you show you know the difference.

Run the disasters, narrating, in this order:

Kill each box, and I mean each. The cache: a thundering herd, where every request that used to hit the cache now hits the database at once, answered with request coalescing (one request fetches, the rest wait for its answer; the caching post's stampede chapter) and a circuit breaker (a switch that stops calling a failing dependency for a while instead of piling on). The database primary: how long does failover take, who notices, and what's the RPO (recovery point objective: how much recently written data you're willing to lose) and the RTO (recovery time objective: how long you're willing to be down)? One API server: nothing happens, and saying "nothing happens" with confidence is itself a signal, because it's the point of stateless servers. A worker fleet that runs out of memory (OOM) and crash-loops: does the queue in front of it hold the work, or drop it? Name the blast radius of every component. The box whose death you can't bound is the box you redesign.

Partition the network, because the network will partition. Two regions that can't talk: which one serves writes? What diverges, and how does it reconcile? Can both regions end up believing they're the primary (split brain), and what stops that? This is where you say the consistency sentence again: reads stay available, money stays correct by construction. The candidate who never mentions partitions is designing for a network that doesn't exist.

Flood the hot path. Ten times the traffic, one key with a million hits a minute, a deploy that retries every failed request twice. What sheds load first, and is the shedding deliberate? A 429 you chose beats a cascading timeout you didn't. This is rate limiting (#8) meeting resilience (#2): limits at the edge, breakers on the dependency paths, queues with bounds, and bulkheads (isolated resource pools, so one overloaded feature can't consume the threads every other feature needs).

Corrupt the data. The disaster nobody puts in the diagram: a bad deploy that writes garbage for an hour before anyone notices. Replication doesn't help, because the replicas copied the garbage faithfully. This is where you need backups you've actually restored from, point-in-time recovery, and a realistic RPO for this failure, which is set by how often you archive the write-ahead log rather than by your replication lag, and where the real loss is the legitimate writes tangled up with the bad ones after the restore point. A related one: a third-party dependency (the payment provider, the email service) goes down for a day. What's the degraded mode? Queue the work? Fail closed? Show the user something you can stand behind?

Then the adversarial turn, the interviewer's meanest five minutes, and yours to preempt:

"What's the worst a user can do with this API?" Ask it of your own design before they do. Sequential IDs get enumerated (the interview's adversarial turn, answered with unguessable IDs and signed URLs whose signature is an HMAC, a keyed hash only your servers can compute; the security post has the recipe). Creation endpoints get farmed (rate limits on the expensive paths). Retry buttons get double-clicked (the idempotency key from Section 4's API sketch; see how the method compounds?). Security isn't a separate chapter of the interview; it's the failure-mode pass with a malicious actor.

One more disaster people forget, because it's not in the diagram: the deploy. New code has to reach production without becoming an incident. Say it: rolling deploys, a few servers at a time; connection draining, so in-flight requests finish before a server is taken out (the load balancer post); a canary, where the new version gets 1% of traffic while you watch its p99 and error rate before the rest; a feature flag as the kill switch, so a bad feature is turned off in seconds rather than rolled back in minutes. "How does code get here safely?" is a design question, and the candidate who answers it unprompted has just separated themselves from everyone who treats the diagram as a static artifact. Systems aren't deployed once. They're deployed weekly, forever.

You don't have to survive every disaster. You have to have an answer for every disaster, including "we accept this one, and here's why."

That last clause matters. "We accept brief 404s for brand-new links in the far region during a partition, because the alternative is blocking all writes everywhere" is a better answer than a hand-waved "we'd handle it." Which failures you accept is a budget decision, and the observability post's error budgets are the formal version of it. Principals choose their outages. Juniors pretend there are none.

Section 8 — The principal round

Everything so far gets you to "hire" at your level. This section is what gets you to "strong hire" at the senior levels, where the questions sit above the boxes: what breaks at 10×, what it costs, how you'd migrate to it, what keeps you up at night, and whether you can say what you sacrificed.

Multiply by ten. Then by a hundred. Take your numbers from Section 3 and multiply. At 10×, 4,000 reads a second becomes 40,000: does the cache tier hold, or does one hot key melt a node? At 10× the storage, does the shard count still work, or is resharding due? Then 100× and beyond: the interview's own jump was a thousandfold, to 4 million a second, and at that scale you're not serving redirects from the origin at all; the answer was the edge (#7). Every design has a scale at which it stops being that design. Naming yours ("this holds to around 50,000 rps; past that, the read path moves to the edge") is the sentence that most reliably marks a principal.

Say the trade-offs in the sentence format interviewers write down. Not "we use async replication," but "I chose async over sync replication because our writes need single-digit milliseconds and we can tolerate seconds of cross-region lag on reads; the cost is brief 404s for new links in the far region during a partition." The format is always: "I chose X over Y because [requirement], accepting [cost]." Say these as you make the choices, during the deep dives, rather than saving them for the end, where there won't be time. Three of them, stated crisply, outweigh twenty minutes of diagram. They're the evidence of judgment.

Price it and page it. Two questions close every senior round worth passing: where's the money going? and what wakes you up? For the money, name this system's two biggest line items with numbers, whatever they turn out to be; sometimes it's the cache cluster and the egress bill, and sometimes compute is the largest line by far, so don't assume. "4,000 redirects a second at a few hundred bytes each is about 2 MB/s of egress, trivial; the cache at a 95% hit ratio means the database sees 200 reads a second, which is one primary with a replica for failover rather than a fleet. If I had to cut 30%, I'd let the cache evict the long tail sooner and run a smaller node, and the math says we'd still hold the p99." Cost answers with numbers beat cost answers with adjectives, and the estimation post is what makes the numbers fast.

For the paging, name three alerts (cache hit ratio, p99 latency, replication lag, queue depth), each tied to a user-visible symptom, with the service-level objectives you derived from the board's non-functional requirements. A design with no cost model and no alerts is a demo, not a system. And ask the operability question of your own design: "who gets paged when this breaks, and do they know what to do?"

Say how you'd get there from here. Real systems are never built on a blank whiteboard; they replace something. The migration has a shape, and giving it in four sentences is a staff-level signal. Dual-write to the old and new stores while you backfill history. Shadow-read from the new store and diff the answers against the old one until the diff is clean. Flip reads to the new store, then writes, keeping the old path alive for a week with a rollback plan. For splitting a service out of a monolith, the strangler pattern: route one endpoint at a time to the new service until the old one is empty. Name who owns each phase.

Revisit consistency per operation, one last time. At 10×, the calls from Sections 6 and 7 get re-examined: is the replication lag still seconds, or has it become minutes, and does the RPO promise still hold? Does the idempotency-key window still cover the longest retry a client might make? Scale doesn't just stress boxes; it erodes guarantees, and the principal re-verifies them.

Close with the question the interviewer asked at the end of Step 11: what would you do differently? "I'd have drawn the failure paths first" was the interview's answer. Yours should be specific to the problem: the box you'd add, the call you'd reverse, the assumption from minute five you'd now challenge. Regret, stated precisely, is the sound of learning out loud, and it's the last thing the interviewer hears.

Section 9 — Who's in the room

Same problem, four engineers. "Design a ride-sharing dispatch: riders request, drivers accept, match them fast." Watch what changes with level: the behavior, not the boxes.

The pattern: each level widens the circle of what "the problem" includes. Junior solves the diagram. Senior solves the requirements. Staff solves the trade-offs. Principal solves the context: the organization, the migration, the sequencing, the question behind the question. You don't fake a level by knowing more boxes. You demonstrate it by including more of reality, and, often, by saying less: principal answers are frequently shorter, because they spend their words on the one thing that matters and name the rest as known.

A note on fairness: the junior column isn't a strawman. Competent juniors ask questions and do arithmetic. The real gap at that level is depth and failure modes, not process. And the ladder is one common shape, not a law; many companies' interview loops top out at staff, and titles vary wildly between them.

You signal level by narrating, not by claiming. Nobody says "I'm thinking at staff level now." They say "the trade-off here is…" and "at 10× this breaks, so…" and "the migration from what you have today would go in three phases…" Level is audible in the sentences, not the boxes. If your answer sounds like the right column of the table, it doesn't matter what the diagram looks like.

And it's a collaboration, so behave like a colleague. Pause every couple of minutes with "does that match what you had in mind?" Take hints the moment they're offered and say "good point, let me change that" rather than defending the old answer. Think aloud without monologuing. Keep the board legible: requirements top-left, numbers top-right, numbered boxes, and a clean corner for the API. The interviewer is deciding whether they'd want you in their design reviews, and a design review with someone who can't take a hint is a bad afternoon.

Six traps to retire before you walk in: boxes before questions (the marker-first candidate from Section 1; every interviewer has a story about one); no numbers (the spaceship to cross the street); happy path only (no failure pass, no adversarial turn); one consistency answer for everything; no deploy or on-call story (a system nobody can ship or page on isn't a design, it's a drawing); and depth everywhere, nowhere (seven topics, one inch deep).

The interview is a simulation of working with you. Act like the colleague you'd want on the worst day: asks first, measures second, decides out loud, and knows what breaks.

Practicing it

The method above is learnable, and it's learned by repetition under time pressure, not by reading. Four things that work:

• Timed mocks with a rubric. Forty-five minutes, a peer holding the six rubric axes from Section 1, and feedback on each. Three or four of these change more than thirty hours of reading.
• The ten-minute drill. Take ten prompts and do only the first two moves for each: clarify and math, ten minutes, no boxes. The first five minutes are where most interviews are decided, and they're the part you can practice fastest.
• Record yourself. Listen back for hedging, for questions you didn't ask, and for the trade-offs you made silently instead of saying "I chose X over Y because." Explaining the design to something that can't answer (a rubber duck, a voice memo) exposes the steps you skipped.
• Practice on the tool you'll use. Many loops are remote now. If the interview is in a shared drawing tool, practice in one, because a board you can't draw on quickly is a board you'll under-use.

The Whiteboard Playbook, distilled

The one-page checklist. Forty-five minutes, six moves. Carry it in.

Minutes 0–5: Clarify. ☐ Functional requirements, listed. ☐ "Out of scope," said explicitly, and the scope cut proposed by you. ☐ Read:write ratio, scale, latency, consistency, availability, durability, cost ceiling: answers written on the board. ☐ "Is the traffic smooth or bursty?"

Minutes 5–10: Math. ☐ Per-second traffic (÷86,400, ×2–3 peak, more for campaigns). ☐ Storage (record × count × growth × replicas). ☐ Bandwidth (bytes × rps). ☐ The ratio named: "this is a ___ problem." ☐ Sanity-checked against an anchor. ☐ Sized to these numbers, with the seam for later named.

Minutes 10–18: Skeleton. ☐ Client → LB → API → cache → DB → queue → workers, plus observability, drawn. ☐ API endpoints sketched: idempotency keys on mutations, cursor pagination on lists, retryable vs non-retryable errors. ☐ Three entities, keys, indexes, shard key. ☐ Storage picked from the access pattern, one sentence of why. ☐ One request traced end to end.

Minutes 18–30: Deep dives. ☐ 2–3 areas chosen by the ratio and the novelty. ☐ Everything else named and parked. ☐ "I chose X over Y because…, accepting…" said as each choice is made. ☐ Toolbox consulted: caching, sharding, IDs, replication, idempotency, async, rate limiting, CDN, LB, resilience, observability, security. ☐ Checkpoint announced at minute 25; if behind, cut and say so.

Minutes 30–40: Break it. ☐ Kill each box; blast radius named. ☐ Partition the network; who serves, what diverges. ☐ Flood the hot path; what sheds first, deliberately. ☐ Corrupt the data; backups, RPO, RTO. ☐ "Worst a user can do": enumeration, abuse, retries. ☐ The deploy: canary, draining, kill switch.

Minutes 40–45: Close. ☐ 10×/100×: what strains, what breaks, the threshold named. ☐ The two biggest cost lines, with numbers, and the lever. ☐ Three alerts, symptom-based, tied to the SLOs from minute three. ☐ The migration in four sentences. ☐ "What I'd do differently," specific.

Traps, taped to the monitor: boxes before questions · no numbers · happy path only · one consistency answer for everything · no deploy or on-call story · depth everywhere, nowhere.

Further reading

• Donne Martin's System Design Primer. The community's open-source companion to everything above, including a worked shortener.
• Google's Site Reliability Engineering books. The failure-mode, error-budget, and alerting chapters are the principal round in prose.
• Jonas Bonér's gist of Jeff Dean's latency numbers. The table from the estimation post, after Peter Norvig's original.
• Martin Fowler: MonolithFirst. The counterweight to premature decomposition; Section 3 reconciles it with sizing for growth.
• Dean & Barroso: The Tail at Scale. Fan-out tail amplification and hedged requests; behind Section 6's latency budget.

Where you'll meet this

Everywhere the other thirteen posts live, because this post is the other thirteen posts, organized into forty-five minutes. You'll meet it in the interview loop, obviously. You'll meet it again in every design review, where "what's the ratio, what did you trade off, what breaks at 10×" is the same playbook with more time and a real budget. And you'll meet it on the day the system you designed is the one paging you, which is the real reason the method insists on failure paths, alerts, and cost before the marker goes down.

The series started with a dictionary on a laptop and ends here, with a procedure for turning any blank whiteboard into a designed system. Ask first, measure second, decide out loud, and break it before the interviewer does.

Keep exploring: the Core Concepts series

Every post in the series stands alone. Read them in any order.

• From Laptop to Billions of Clicks: Designing a URL Shortener in 11 Steps — the anchor: one design, every concept under load.
• #1 The 100:1 Superpower: Caching — the fastest request is the one you never make.
• #2 The Blast Radius: Surviving the Day Your Dependencies Fail — staying up when everything you depend on goes down.
• #3 Divide and Conquer: Sharding and Partitioning — splitting one database into many without losing your mind.
• #4 The Snowflake Problem: Unique IDs at Scale — naming things when millions are born every second.
• #5 Copies of the Truth: Replication — keeping copies of your data that actually agree.
• #6 Do No Harm Twice: Idempotency — making "just retry it" safe.
• #7 The Copy at the Doorstep: CDNs and Edge Computing — serving from next door instead of across the ocean.
• #8 The Bouncer's Math: Rate Limiting — saying no politely, at scale.
• #9 What Broke at 3 AM: Observability, p99, and Useful Alerts — knowing what's wrong before your users tell you.
• #10 Do It Later, On Purpose: Async Processing and Queues — the work the user doesn't have to wait for.
• #11 The Traffic Cop: Load Balancing — the box in every diagram nobody explains.
• #12 Assume They're Already Knocking: Security and Abuse at Scale — designing for the users who are designing against you.
• #13 Do the Math First: Estimation for System Design — the two minutes of arithmetic that choose the architecture.
• #14 The Whiteboard Playbook: Taking On Any System Design Challenge — the whole series in forty-five minutes. (this post)

Blueprints of Scale — Core Concepts #14. If this helped, the best thanks is a share with someone who's learning.