Peer Failover: The Redundancy That Manufactures Its Own Outage
Google’s chapter on cascading failures describes a small and very familiar disaster. One cluster fails, and its traffic moves to the next. The next one cannot carry the combined load, and it starts to fail as well. Nothing in that story is unusual, because it is simply what happens when spare capacity turns out to be busy. In the first edition of this newsletter I promised to come back to the topology that makes this most likely, where cells are paired so that each one covers another. This is that piece, and it comes in three parts: why the design is so attractive, the arithmetic that undoes it, and what to build instead.
The design that is easy to like
The pairing idea is an attractive solution to provide redundant coverage for a given set of cells, as each cell is backed up by a partner cell. The layout is symmetric, and no capacity sits idle. From a cost perspective, the backup capacity looks inexpensive, because the partner is a cell that is already running, so there are no extra deployments and the bill looks the same as before. On a whiteboard it reads as redundancy for free.
The question that breaks it
However, from a logical standpoint, the design has to address the fact that both cells will be running during peak loads, since their peak-load periods tend to overlap. When the failure arrives, the partner is serving its own traffic, usually at the same peak. While pressure-testing a design like this, I was tracing the flow of traffic of cell A, which suddenly died due to peak load, and now the traffic was being switched to cell B. It did not take long to see where this approach was going to land.
The arithmetic
So let us define three cells, with invented numbers. Cell A has 40 units of load at peak, cell B has 30 units at peak, and cell C has 20 units at peak. All three cells are provisioned with 25% more capacity than they need to handle their own peak-load demand. For example, cell A has 50 units of capacity to meet its peak demand, and cells B and C have approximately 38 units and 25 units respectively. These capacities are reasonable for typical daily operations. Let us say these three cells are configured in a ring, where cell A is backed up by cell B, cell B is backed up by cell C, and cell C is backed up by cell A. When cell A fails at peak, all of its load transfers to cell B. Cell B is then asked to carry a total of 70 units of load with only 38 units of capacity, and since it is already carrying its own peak-load demand, it immediately fails as well. Cell B’s load, which now includes cell A’s, moves on to cell C. Unfortunately, cell C is asked to carry 90 units with only 25 units of capacity. The result is a large load spike that continues to walk around the ring. A strict pair, A with B and nothing else, ends sooner but no better, because once B sinks there is nowhere left to go.
Symmetry is the problem, not the feature
While symmetry may look appealing in terms of the design (every cell is identical and no cell is idle), it is that very symmetry that causes the design to fail. Redundancy represents spare capacity that is available on the day it is needed. However, a peer-based design uses this spare capacity to carry normal traffic on a day-to-day basis, rendering the redundancy non-existent in practice. There is a well-documented alternative discipline, which AWS calls static stability. Static stability involves provisioning services with sufficient spare capacity in advance. As such, even in the event of a failure, the service remains operational without having to deploy additional resources. The key aspect of static stability is that the spare capacity must still be unused, and so available, on the day that it is needed.
One fallback, never a peer
Statically stable designs are less elegant than peer-based designs. Instead of configuring each cell as a backup for another cell, every cell fails over to one dedicated fallback. When a cell fails, its traffic flows into the fallback, which takes on one failed cell at a time. The fallback receives no normal traffic of its own, and therefore it is never a peer. Once a fallback begins receiving normal traffic, it becomes a peer and can no longer function as a backup for another cell. Adding cells to a peer-based design results in changes to every cell within the group, whereas adding a cell to a statically stable design is a local change, because the new cell fails over to the same fallback and nothing has to be re-paired.
Size it to the largest cell, not the average
Cells vary in size. Therefore, the fallback must support the size of the largest cell plus some amount of headroom. Sizing a fallback to match the average size of the cells will lead to a scenario where the fallback fails to support the largest cell on the day that it fails. Comparing costs fairly, with N cells of roughly equal size, giving every cell a peer’s worth of spare capacity means buying about one extra cell’s worth of capacity N times over. A statically stable fallback at approximately 1.5 times the size of the largest cell buys this redundancy only once. Choosing the correct headroom for the fallback is somewhat subjective, and it can be estimated by calculating the additional traffic the fallback might serve on behalf of another cell. This can be calculated from the infrastructure required to serve a given level of traffic. However, I would recommend adding some headroom beyond those calculations, because systems rarely operate as cleanly as mathematical models predict.
What one fallback gives up
It would be misleading to stop here. A single fallback represents a single point of failure on the failover path, although not on the primary path. Additionally, a failure that affects several cells simultaneously, through a common network for instance, is larger than any single fallback can absorb. The biggest drawback of a single fallback is that it can fail when two or more cells fail at overlapping times, overwhelming the capacity that the fallback was designed to provide. Nevertheless, I still think this model is acceptable, so long as each cell’s failure occurs independently of all the other operating cells. Correlated failures are an entirely separate problem with a different solution, and pretending that one fallback solves them would be worse than admitting it.
When peering is actually fine
Before completely dismissing peer-based designs, they deserve a fair hearing. If every cell operates at significantly below half of its maximum capacity, a peer can absorb its neighbour’s load. Furthermore, if load is shed rather than moved during overload conditions, there is essentially nothing for the peer to absorb.
Who should own it
Since a statically stable fallback appears idle on any cost report, someone needs to own this rule and be able to override those reports. That is typically the person who is responsible for meeting the high-availability targets and who also signs off on the risks the design accepts, such as a correlated failure larger than one fallback. Specifically, ownership rests with the person in technical leadership who is accountable for uptime, rather than the capacity planner. The rule is simple: the fallback should never serve normal traffic, regardless of who requests it or authorises it. Further, whenever the largest cell grows beyond current expectations of capacity, the fallback must be resized accordingly, not when the budget cycle comes round.
Elegance in resilience design generally implies that spare capacity is being reused, and reused capacity is precisely what is missing when it matters most. If managing an additional cell adds so much expense that organisations choose to ignore it altogether, wait for the moment when one of the cells dies and affects the business, with lost revenue climbing and customer satisfaction falling. But ultimately such decisions still depend upon how critical the business is.
Which of your cells is somebody else’s fallback, and how busy is it at its own peak?