A Database in a Single Zone: What the 20 October Outage in London Teaches About High Availability
A Google Cloud zone in London lost power for six hours, and zones that kept their power were affected too. A database's high availability is not proven by the architecture diagram: it is proven by a timed zone failover.
On 20 October, a Google Cloud zone in London lost power for six hours and ten minutes. The grid failed, the batteries held the load, and the generators did not start. Two automation systems existed to start them, the primary and the backup. Neither gave the order.
The case is about a data centre, but the question it raises is about architecture: where does the primary database run, and what happens when that zone disappears for an entire morning?
What happened on 20 October
According to Google Cloud's preliminary incident report, the incident took place in the europe-west2 region (London) between 07:00 and 13:10 London time, a total of 6 hours and 10 minutes. A power grid failure hit the data centre that hosts zone europe-west2-b. The load moved to the batteries while the diesel generators were starting.
The generator automation system, both primary and backup, failed to start the automatic transfer. When the battery reserves ran out, the physical servers, the network and the local storage in the zone lost power and went offline.
The report should be read for what it is: preliminary. The final report may change details of the cause, and this article will be updated if it does.
The detail that matters: impact outside the zone
The same report says that some customers with workloads in zones that did not lose power, europe-west2-a and europe-west2-c, also experienced impact. The causes given were sudden demand surges on the compute capacity of the healthy zones and control-plane validation dependencies that cross zone boundaries.
At the same time, the report states that Cloud SQL, AlloyDB, Memorystore for Redis and Cloud Bigtable failed over, during the incident, into nearby zones that were not affected. Some databases therefore changed zone and kept working.
Two readings follow. The first is that a replica in another zone makes a difference. The second is that a replica in another zone does not exempt anyone from impact: when a zone fails, the others receive the load that no longer fits in the first, and they share control services with it. This second point is xGrowth's reading of the report, not a conclusion from Google.
Zone, region, provider: three levels of failure
A database can fail at three scales, and each calls for a different response.
- Zone failure. A zone becomes unavailable, as in London. The response is a replica in another zone of the same region and a tested promotion process.
- Region failure. Several zones, or the whole region, are affected. The response is a replica in another region, with the cost and latency that implies.
- Provider failure. The problem crosses regions or sits on the other side of the contract. The response is a copy outside the provider, which allows the service to be rebuilt elsewhere.
The article on cloud provider outages and disaster recovery deals with the third level. London forces a look at the first, which is the most frequent and the most underestimated: a database running in a single zone has nowhere to fail over to.
Why almost nobody tests
Having a replica drawn on the diagram is not the same as knowing how long it takes to use it.
The London case is the large-scale version of the same habit. The generators and the two automation systems existed. The report does not say when the controllers were last tested, and this article does not claim that they were not. What the report shows is that existing and working when needed are different things.
How to start testing
A zone failover test does not require new tools. It requires a window and a stopwatch.
Start outside production. On a copy of the environment, shut down or isolate the zone where the primary database runs.
Time it from start to finish. From the moment the zone goes down until the application serves requests again. That number, measured and not estimated, is the real recovery time (RTO).
Check the application. Promoting the replica is not enough. The application has to reconnect to the new database on its own, without someone editing a configuration at three in the morning.
Write down what failed. Every surprise in the test is a fix that will not have to be discovered during a real outage.
Repeat before each load peak. For retail, the next one is Black Friday, on 27 November. A little over four weeks remain, enough time for a first test.
Frequently asked questions
Does a replica in another zone guarantee there is no downtime? No. It reduces the risk of the database being unavailable when a zone fails, but there is always a promotion time, and the London report shows that neighbouring zones can also be affected. The guarantee only exists once the time has been measured.
Is a second region needed? It depends on what an hour of downtime costs for each service. For many workloads, a tested replica in another zone covers most of the risk. A second region is justified when the cost of a regional failure exceeds the cost of maintaining it.
Who should decide the zone failover? Someone with authority, and a named deputy, appointed before the incident. A change that requires convening a committee in the middle of an outage loses the time the replica was meant to save.
Conclusion
The 20 October outage hit one zone and still had consequences outside it. The infrastructure's redundancy existed, but it did not work when needed. The same can happen with a database replica that has never been used on purpose.
To measure the real recovery time of the primary database and prepare the first zone failover test, it is possible to book a Clarity Session, with no commitment, on systems reliability (SRE) and database reliability (DBRE).