Designing for a Lost Data Center: Multi-Region Failover When the Facility Itself Is Attacked

Most failover guides quietly assume the building is still standing. A bad deploy, a dead availability zone, a cut fiber: you move traffic, wait a few minutes, and go back to bed. Those are the failures the documentation was written for. A facility can also be physically gone. A drone can hit it, a crew can cut its power, or someone who got into a controller can switch it off. Then the question changes. It is no longer how fast you fail over. It is what your system does when the region it leaned on is gone for weeks and its own management tools cannot be trusted.

Why now: Between 8 and 11 October, according to BleepingComputer, drone strikes damaged three Yandex data centers. Two stopped completely. Yandex’s status message says the platform is in emergency mode and recommends “any other alternative disaster recovery plans.” Separately, BleepingComputer reports that attackers used the built-in defrost function of refrigeration controllers to warm stock at French supermarkets.

Table of contents

• Designing for weeks, not seconds
• Headroom: the bill you pay before the outage
• Cells: where the blast radius stops
• Stop asking the dying region for permission
• DNS timing you can add up
• Replication lag is your recovery point
• The shared plane nobody draws
• A runbook you can copy
• Where this design breaks
• Did you know
• Sources

Designing for weeks, not seconds

Most resilience material trains you for a blip. Pick a number of minutes, rehearse a flip, measure the recovery time. Yandex’s case is the opposite. Sasovo and Vladimir stopped operating completely, Kaluga is degraded, and the company calls the remaining resource configuration “unstable.” It has not given a restoration date, and it says it cannot offer a reliable estimate, even with teams working around the clock.

That changes the sizing rule. If a surviving region has to absorb a dead region’s traffic, the capacity has to exist before the outage. Adding machines mid-incident means calling a control plane, and the control plane may be the part that broke. AWS’s static stability guidance says systems should “continue to operate as normal without the need to make changes during the failure or unavailability of dependencies.” It also states that control planes are “statistically more likely to fail than data planes.” Every on-call engineer has tried to load a status page during an outage, only to find it hosted on the thing that broke.

Headroom: the bill you pay before the outage

Here is the arithmetic I would tape to the wall. Assume three active regions, a combined peak of 100,000 requests per second, and a rule that no region should exceed 70% utilization after a failover. These are my assumptions, not AWS’s. Swap in your own numbers.

	Three regions	Two regions
Normal load per region	33,333 req/s	50,000 req/s
Load per survivor after losing one region	50,000 req/s	100,000 req/s
Capacity per survivor at a 70% target	71,429 req/s	142,857 req/s
Total provisioned	about 214,286 (2.14× peak)	about 285,714 (2.86× peak)
Normal utilization per region	about 46.7%	35%

More regions spread the spare capacity thinner. The two-region setup sits at 35% utilization every day to cover a failure that might never come, and it costs more in total than the three-region one. That is the insurance premium in plain sight.

Cells can cut the bill further. If each region is split into cells and only the failed cell’s tenants move, the survivor needs headroom for that share alone. I have not found a published figure for that, so model it with your own cell sizes. Pre-provisioned capacity is still a budget argument, and nobody outside your organization can tell you where the line falls. I would still rather pay that bill than discover in week two that the failover target is also short on capacity.

Cells: where the blast radius stops

A cell is a full copy of the stack that serves a fixed subset of tenants. A thin router maps each tenant to one cell, so a failure stays inside that cell. AWS’s cell-based guidance says routes to cells are “loaded into memory from an S3 bucket,” and that “even if the control plane, Amazon S3 or a zone is unavailable, the router is still able to direct traffic to the cells.” The router keeps working because it reads its map from memory instead of asking a service on every request.

The design rule that follows: nothing on the request path should call a service that dies with the facility. Then failover means repointing a cell’s endpoint to its surviving copy, a much smaller change than rebuilding a stack during an incident. Keep the routing map small, version it, and make sure every region loads the same snapshot. Otherwise your isolated cells are only isolated until someone checks which map is live.

Stop asking the dying region for permission

A failover that waits on the failed region’s API is a fire exit that needs the building manager to unlock it. The exit exists. The unlocking does not.

AWS’s Application Recovery Controller (ARC) is built around that problem. Its routing controls are, in AWS’s words, “simple on-off switches hosted on highly available clusters spanning five AWS Regions,” and they integrate with Route 53 health checks. A cluster is five endpoints, one per Region, which AWS says work in coordination “so that they guarantee high availability and sequential consistency of failover operations.” The limit to plan around: “Each account can have a maximum of 2 clusters.”

Setup takes three CLI calls. Fill in the placeholders:

# 1. Create the cluster. Status goes PENDING -> DEPLOYED; poll describe-cluster.
aws route53-recovery-control-config create-cluster \
  --cluster-name test --network-type DUALSTACK

# 2. One routing control per cell. The cluster ARN is required.
aws route53-recovery-control-config --region us-west-2 create-routing-control \
  --routing-control-name RoutingControlCell1 \
  --cluster-arn <cluster ARN>

# 3. Route 53 health check that reads the routing control's state.
aws route53 create-health-check \
  --caller-reference RoutingControlCell1 \
  --health-check-config Type=RECOVERY_CONTROL,RoutingControlArn=<routing control ARN>

Two gotchas from AWS’s CLI guide. Creating a cluster also creates a DefaultControlPanel, and a control panel can live in only one cluster, so moving it means deleting and recreating it. Second, AWS’s failover example puts one record at PRIMARY and one at SECONDARY, each with a HealthCheckId, and sets TTL to 0 on both. The page does not say that 0 is what you should use. Decide that on purpose rather than copying it.

Then add a safety rule so automation cannot take down every cell. AWS’s assertion-rule example keeps at least one of two controls On, with a threshold of 1 and a 5000 ms wait period. If a change breaks a rule you created, the call throws an exception, which is exactly what you want at 3 a.m.

The flip itself is where I have to stop short. The guide names set-routing-control-state in its safety-rule text, but I could not confirm the exact data-plane command from the pages I reached. Check the CLI reference before you paste it into a runbook.

ARC also offers two ways to move traffic away from an impaired zone. Zonal shift is manual: “Zonal shift is manual—you start and stop it.” Zonal autoshift is the opposite bet: “AWS detects impairments and shifts traffic on your behalf.” Manual keeps the decision with you and costs human minutes. Autoshift removes the human delay but makes your failover depend on AWS’s detection. AWS’s own autoshift practice runs last “approximately 30 minutes,” a reminder that rehearsals are not instant either. Shifts expire too, anywhere “from 1 minute up to 3 days (72 hours),” so decide in advance what happens when one lapses in the middle of an outage that is still going.

DNS timing you can add up

Worst-case client cutover is roughly the detection time, plus the record TTL, plus any resolver that caches past the TTL. Health checks give you the first term with published numbers. Route 53 checkers request your endpoint every 30 seconds (standard) or every 10 seconds (fast, which costs extra), and the interval cannot be changed after creation. AWS defines the failure threshold as “the number of consecutive health checks” an endpoint must pass or fail before its status changes. I could not find the default value in the material I reached, so the timeline below assumes three consecutive failures.

Health check interval	Detection (3 failures, assumed)	Plus 60 s TTL (my example)	Worst case, before resolver misbehavior
10 s (fast)	about 30 s	60 s	about 90 s
30 s (standard)	about 90 s	60 s	about 150 s

Those figures are the floor, not the whole story. Route 53’s checkers do not coordinate, and AWS warns you’ll “sometimes see several requests per second regardless of the interval you chose, followed by a few seconds with no health checks at all.” My own estimate adds up to one more interval of phase offset, which AWS does not state. And a resolver that ignores TTL is outside your control, so the TTL is a request, not a guarantee. The 60-second TTL is an example, not a recommendation.

Two more traps. Calculated health checks can combine up to 256 children with AND, OR, or “at least x of y,” which is how you encode “two of three probes agree.” And if you health-check a CloudWatch alarm instead of an endpoint, watch the default: a new health check with no last known status starts out healthy. A metric that goes quiet during a facility loss can therefore look fine. AWS advises against relying on last known status when the metric is unavailable for more than a few hours.

If you drive failover from a routing control instead, detection becomes an operator or automation action, plus ARC propagation time. I could not find that propagation time documented in what I reached. Measure it in a game day, because it goes straight into your recovery time.

Replication lag is your recovery point

DynamoDB global tables replicate writes “usually within one to two seconds,” and AWS does not promise a hard bound. The word “usually” carries the sentence. On a sudden facility loss, the writes you lose are roughly the ones still in flight during the lag, plus anything not yet replicated across Regions.

Measure that number instead of assuming it. DynamoDB publishes ReplicationLatency for global tables, in milliseconds, with the dimensions TableName and ReceivingRegion. The metrics page puts five-minute aggregation on every metric that is not in its one-minute list, and ReplicationLatency is not on that list, so I read its granularity as five minutes. Enter the alarm as follows in the console, or translate it into your infrastructure-as-code:

Setting	Value
Metric	ReplicationLatency (Milliseconds)
Dimensions	TableName, ReceivingRegion
Statistic	Maximum
Period	5 minutes (matches the metric’s granularity)
Threshold	Your RPO in milliseconds. AWS publishes no recommended value.

The five-minute period has a cost. An alarm needs at least one full period before it can fire, so treat it as an early warning for the operator, not the trigger for failover. Let your health checks and routing controls do the switching.

Replication mode decides what a sudden loss costs you, and PostgreSQL’s documentation makes the trade-off unusually explicit:

Replication mode	What you can lose on sudden primary loss
DynamoDB global tables	Writes still in flight during the lag. Usually one to two seconds, with no hard bound.
PostgreSQL asynchronous (the default)	Committed transactions not yet received by the standby. Loss is “proportional to the replication delay at the time of failover.” Lag is typically under one second when the standby keeps up.
PostgreSQL synchronous	Only if the primary and the standby crash at the same time. Each commit waits for the WAL to reach disk on both.
PostgreSQL ANY 2 (s1, s2, s3)	Commits wait until two of three standbys hold the WAL. On my reading, one failed standby does not lose acknowledged commits.

The PostgreSQL docs add a caveat that matters for any quorum design: “There is no way to be certain that all standbys have received all outstanding WAL data at time of the crash of the primary.” Standbys can disagree about what was committed. Asynchronous replication is fast and quietly lossy. Synchronous and quorum modes are slower and bounded. DynamoDB sits in the first camp, which is a fine place to be if you know the number.

Writes can collide too. When two regions accept writes to the same item, DynamoDB resolves the conflict “by using a last-writer-wins resolution method.” That is simple and always converges. It can also drop a write without raising an error. Picture a later write in the dying region and an earlier one in the survivor, both inside the lag window. At failback, the later timestamp wins, and the survivor’s write is gone with no error to find.

The cleaner fix is to stop letting two regions write the same item. Pin each tenant’s items to a home region, using the cell map from earlier. Conflicts then appear only during failover, when a survivor takes over writes. That moves the risk into a window you can design, test and alarm on, which beats discovering it at failback. It does not remove the risk, and it means your failover runbook now owns the consistency problem. For data that must merge instead of overwrite, look at conflict-free replicated data types (CRDTs). I did not verify their mechanics in this pass, so treat that as a pointer, not a design.

The shared plane nobody draws

The French case is a different failure, and it matters for cell design. BleepingComputer reports that attackers hijacked controllers running older Danfoss software and used the built-in automatic defrost function to raise temperatures. The named chains include Intermarché, Coopérative U, Carrefour Market, Monoprix, Leclerc and Super U, and French outlets put the count at 250 to 350 stores. Danfoss is reported as the target, not confirmed, and it did not comment. The Paris prosecutor’s office has opened an investigation, and no suspects have been identified. Coopérative U’s president said a cyberattack “has been affecting the refrigeration systems in several stores” and called it “unacceptable.”

Nothing here needed a fire or a cut cable. It needed a remote-management plane that many sites trusted in the same way, and a defrost feature that works just as well when someone else presses the button.

BleepingComputer cites a CISA warning about two vulnerabilities in the Danfoss AK-SM 8xxA platform, CVE-2025-41450 (authentication bypass) and CVE-2025-41451 (command injection), and reports no evidence linking them to this attack. Predicta Lab counted 2,752 Danfoss systems exposed online, 1,590 of them in France.

For cells, the lesson is blunt. A cell boundary drawn along availability zones protects you when a zone goes dark. Against a shared vendor, a shared firmware version or a shared remote-management credential, it does nothing. If every site in every cell runs the same controller with the same admin login, your isolated cells are one cell with extra steps. Two rules follow. Draw cell boundaries on management credentials as well as network placement. And record the firmware version as part of each cell’s identity, so you can see when many cells run the same build.

A runbook you can copy

1. Detect with several signals. Combine probes with calculated health checks, using AND, OR or at-least-x-of-y. Write a degraded branch (Kaluga’s situation) and a fully-down branch (Sasovo’s and Vladimir’s) as separate paths, since they call for different responses.
2. Shift in one reversible step. Flip the routing controls for the impaired cells toward the survivor’s cells. Keep your safety rule in place, log the timestamp, and make sure the change can be reversed.
3. Verify before you trust. Check cell routing and ReplicationLatency in the surviving region before sending full traffic. If lag is climbing, you are not ready to accept every write.
4. Set the shift expiry in advance. If you use zonal shift, choose an expiry that will not lapse in the middle of an outage that is still going.
5. Hold failback. Do not return traffic because the lost region looks healthy. Wait until the facility is verified and replicas have caught up, or you trade one failure for silent conflict losses.

Keep the timing math on the same wall as the headroom math. Worst-case cutover is detection, plus TTL, plus whatever resolvers ignore. Each term is a lever you control only partly.

Where this design breaks

• Yandex’s public statements say nothing about how its systems were architected. We know the outcome, not the design, so I cannot grade its failover.
• ARC and autoshift change whom you trust. You stop depending on a dying region’s control plane and start depending on AWS’s clusters and telemetry. That is a better bet, not the removal of a dependency.
• Pre-provisioned headroom turns a risk into a line item. That line item is expensive and easy to cut in the next budget cycle, right up until the day it is needed.
• Silent conflict loss is the failure nobody alarms on. Pinning moves it into the failover window, where you can test it, but it does not remove it.
• I could not find published numbers for the full chain, from detection through TTL, caching and replication lag. You will have to measure yours. That measurement is the real work, and no vendor page does it for you.

If I had to pick one first step, it would be the least glamorous: write down the headroom the surviving region needs, and pay for it before the bad day.

Did you know

• Drone strikes damaged three Yandex sites between 8 and 11 October, according to BleepingComputer. Two stopped completely. The third is degraded, which is the case most runbooks forget.
• An ARC cluster is five endpoints, one in each of five AWS Regions, and an account can have at most two clusters.
• A zonal shift can be set to expire anywhere from 1 minute to 72 hours.
• Route 53 checkers do not coordinate, so you may see several health-check requests in one second, then a few seconds with none.
• Predicta Lab counted 2,752 Danfoss systems exposed online, 1,590 of them in France.

Sources

• BleepingComputer, “Yandex confirms service outage as users lose access to cloud files”: https://www.bleepingcomputer.com/news/security/yandex-confirms-service-outage-as-users-lose-access-to-cloud-files/
• BleepingComputer, “Cyberattack shuts down refrigerators at French supermarkets” (also the source of the CISA and Predicta Lab figures): https://www.bleepingcomputer.com/news/security/cyberattack-shuts-down-refrigerators-at-french-supermarkets/
• AWS, Fault Isolation Boundaries whitepaper, “Static stability”: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/static-stability.html
• AWS Well-Architected, “Cell-based architecture: control plane and data plane”: https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/control-plane-and-data-plane.html
• AWS, Application Recovery Controller FAQs: https://aws.amazon.com/application-recovery-controller/faqs/
• AWS, “Set up routing control components” (ARC CLI guide): https://docs.aws.amazon.com/r53recovery/latest/dg/getting-started-cli-routing-config.html
• AWS Route 53, “Values that you specify when you create or update health checks”: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/health-checks-creating-values.html
• AWS Route 53, health checks overview: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover.html
• AWS, DynamoDB global tables: https://aws.amazon.com/dynamodb/global-tables/
• AWS, DynamoDB metrics and dimensions: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/metrics-dimensions.html
• PostgreSQL 18 documentation, “High Availability, Load Balancing, and Replication”: https://www.postgresql.org/docs/current/warm-standby.html