Blast Radius Is Not Disaster Recovery : Designing for Self-Inflicted Failure
The book on site reliability engineering that Google published in 2016 reports that roughly 70% of outages are due to changes in a live system. Over years of experience, I have noticed that hardly any failure was due to a cloud provider having massive failures in its data centres, or an outage like the ones Cloudflare has had. Most of the time production issues were due to more local decisions and actions. So, while designing the resiliency, I try to solve something which is the majority of the problem. Also, it is something that is harder to justify and faster to implement than something that is due to the fault of the cloud provider. The argument comes in four parts: why three different problems hide behind one word, why the standard disaster-recovery playbook makes the commonest of them worse, what containing that problem requires, and why it is so hard to fund.
Three problems, one word
If you ask a team to describe its high-availability strategy, the answer is usually some mix of replicas, standby databases and automatic failover between data centres. If you then ask what its last five outages actually were, the list looks rather different. “Resilience”, “high availability” and “failover” are used for three problems that have very little to do with each other. The first is infrastructure failure, where the provider loses a data centre, which is external and rare. The second is demand exceeding capacity, which is a load problem and has a load problem’s answers. The third is damage that is self-inflicted or local, such as a bad deployment, a disk that fails on one host, a network partition, or an operator running the right command against the wrong environment. Public post-mortems show the third one again and again. In 2017 a mistyped input to an operational command took far more Amazon S3 servers offline than intended. In the same year GitLab lost several hours of database records when a data directory was wiped on the primary instead of the secondary.
Designs for peak load and failover are standard now with many strategies already war-gamed, tested and lessons learnt, but designs that cater to the third problem are the ones customised to specific applications and there is no one strategy that fits all. A design that is honest about this says so in its scope. It leaves the first problem to a separate disaster-recovery plan, and it treats the second as a scaling problem rather than as a reason to fail anything over.
Why the disaster-recovery playbook makes it worse
While pressure testing proposed architectures over the years, I realised that replication or a hot standby does not protect against the transient issues that have just started growing numbers, although very good otherwise for high load or hardware failures. This led me to go back to the drawing board. The reasons are mechanical, and four of them can be given.
First, synchronous replication is an amplifier for self-inflicted damage. Replication can backfire when it copies a bad write to a standby, propagating the fault instead of isolating it, and synchronous replication does that at wire speed. In fact, MySQL’s own manual gives protection against “user mistakes on the source” as a reason to delay a replica on purpose. That tells you what an ordinary replica does with a mistake. Second, a hot standby runs the same release as the primary, so the defect is already waiting there. The system still appears healthy and its metrics look normal, but primary and standby are both compromised, and failing over only moves the traffic from one broken copy to another. Third, automatic failover triggered by health checks has nowhere useful to go when every replica is equally sick. Fourth, autoscaling answers the extra load of a misbehaving release by adding more copies of that release. Each of these is the right answer to the first problem, and each of them makes the third problem worse.
Then there is the idea of failing over for peak. Failing over under load does not reduce the load. It moves all of it, at the worst possible moment, onto a target that has its own traffic to carry, and an overload in one place becomes an outage in two. It is common for a sudden failover during a traffic spike to send a flood of requests to a backup that lacks the capacity for the surge. The result is a total outage, worse than the degraded state it was meant to escape. Google’s chapter on cascading failures describes the same thing. One cluster fails, its traffic moves to the next, and the next one cannot take the combined load and starts to fail as well. So load should be scaled rather than failed over, and a cell that is overwhelmed by traffic alone should not trigger a failover at all. In a later edition I will come back to the topology that makes this trap most dangerous, where cells are paired so that each one covers another.
What containment requires
Containment requires a clear partition between cells, so that nothing is shared and a failure in one cannot affect another. Each cell is a complete stack with no database or message broker in common with any other. AWS’s guidance on cell-based architecture says its own service teams have built this way for more than a decade. Partition is only the first of four properties, though. If any of them is missing, whether through shared state between cells, a simultaneous deployment or an automated abort, the system cannot reliably contain a bad release.
The second is staggering, which is the property most likely to be missing in practice. Cells contain a bad deployment only if a release goes to one cell, is watched, and only then moves to the next. If a release goes to every cell at once, the organisation has paid for cells and bought nothing, because the same defect arrives everywhere at the same moment. Cloudflare and CrowdStrike both learnt this in public. In 2019 a single rule change at Cloudflare went to its whole network within seconds, because rule changes were allowed to skip the staged rollout that the rest of its software went through. For Cloudflare’s customers that was an outage at the provider, but inside Cloudflare it was a self-inflicted one. In 2024 a faulty content update from CrowdStrike crashed Windows machines across the world, and the company’s own review committed to “a staggered deployment strategy, starting with a canary deployment”.
Staggering has a cost of its own, and it is better to say so. Staggering releases cause the team to document the specific stage of deployment, test every step, maintain multiple configs and while debugging check what is the state of application in that specific case. Those costs are paid on every release, while the protection is collected only on the bad ones.
The third property is an asymmetric fallback. A fallback must run a known-good version, never the broken release, so that there is a safe way out, and a hot standby fails this test by definition. The fourth is a human trigger. A person, not an automated health check, decides when to abandon a cell and switch to the fallback. The failures in this class so often look healthy from outside, and a probe would mistake noisy metrics for good health. A manual switch means recovery takes minutes rather than seconds, and it can be said that this is a fair price for not letting a probe decide.
Isolation comes from direction
Data should flow one way, from each cell to the fallback and never the reverse, so that bad state from a failed cell cannot contaminate the good version. A fallback has to be warm to take over quickly, which means it needs a steady supply of reference data and of the work in flight in every cell. One-way asynchronous streams provide both. When a cell fails, its streams simply stop and the fallback’s view of that cell falls a few seconds behind. Nothing else happens, because no write travels the other way. Two-way replication works the other way round, since the channel that keeps two places in step is also the channel that carries damage from one to the other. On a whiteboard, the design is a row of isolated boxes for the cells, each with an arrow to a separate safe box for the fallback. There are no arrows at all between the cells. That picture is the requirement: a fault stays confined to its cell, every recovery path is independent, and acting on a failed cell never puts a healthy cell at risk.
The same reasoning argues for a single fallback that every cell fails over to, with no cell ever serving another cell’s traffic. A peer would have to carry a second cell’s load on top of its own, while a fallback is provisioned for one job and takes on one failed cell at a time. Cells differ in size, so the fallback is best sized at around one and a half times the largest of them. Sizing it against the average would fail on the day the largest cell is the one in trouble. The natural candidate for that fallback often already exists. It is the infrastructure that carried the platform before it was split into cells, and it tends to stay around afterwards with almost no traffic on it.
Containment in time, before containment in space
A fast, automated rollback is containment in time. It limits how long a bad release can do damage. The same chapter of Google’s book that gives the 70% figure puts rolling back changes safely next to progressive rollouts, at the centre of change management. Cells are containment in space, and their real job is to protect against the damage a rollback cannot undo, such as corrupted data or money that has already moved.
That said, I think cells are not the first thing most teams should buy. Fast rollback is easier to implement, less duplication of cloud infra, less moving parts and no need to worry about the availability of required data across cells. Many teams should buy containment in time first and then stop, because cells earn their cost only where a few minutes of a bad release can do harm that rolling back will not repair.
Why it is so hard to fund
Containment is harder to fund than protection against the provider, because its benefits stay invisible until something fails. In a budget discussion the room tends to focus on tangible costs, such as infrastructure and the provider’s service levels, and next to them containment looks like an abstract risk expense. The conversation turns to immediate return, and the proposal is labelled “nice to have” rather than essential. Without a recent incident to point to, the upfront spend is hard to justify, even though the long-term saving is large.
There is also a reason that nobody says out loud. “The provider might lose a data centre” accuses nobody in the room, while the sentence that justifies containment, that the team will one day deploy something broken, accuses almost everybody in it. The framing that tends to work better is the speed of delivery. Cells and staggered releases let a team ship often without each release being a bet on the whole system, and nobody feels accused by an argument about speed.
Who should decide
The decision underneath all of this is what the resilience budget is for. It should be owned by the product leader together with engineering leadership, because between them they understand both the business impact and what is technically feasible. The rule I would hold them to is that the money for containment follows the risk exposure of each service. Recent outages are classified by cause before anything is funded, and no release ever reaches every cell at once. The first sign of the rule slipping is a cost-cutting meeting that starts trimming the number of cells, a quiet drift away from the containment strategy. The second is a release that goes out everywhere together “just this once”.
Designing an application or a service is not only building shiny features but it is also giving the user confidence that they can use it any time with 100% confidence that it will be available day in day out. In the end, the only thing that makes your application stand out is the experience of the user. Think of it when the user couldn’t login to the application when it was most needed. Which of your last five outages would your current resilience spending actually have prevented?