Running multi-day AZ evacuation drills with ARC Zonal Shift

This article demonstrates how to conduct multi-day Availability Zone evacuation drills using AWS ARC Zonal Shift to validate sustained resilience across ECS, EKS, RDS, and Aurora databases.

• Multi-day drills expose time-dependent failures missed by brief tests: auto-scaling tuning, stale DNS, connection pooling, and operational processes.
• ARC Zonal Shift removes load balancer IPs from DNS and blocks cross-zone routing to shifted AZs while preserving infrastructure for rapid recovery.
• For ECS: enable zonal shift on ALB, initiate shift, restrict task placement to healthy AZs, and monitor task distribution.
• For EKS: activate zonal shift on cluster, shift both NLB and cluster resources, verify EndpointSlice updates remove pod endpoints from shifted AZ.
• For RDS Multi-AZ: force failover if primary is in evacuated AZ; standby automatically recreates in shifted zone.
• For Aurora: failover writer to reader in healthy AZ; optionally delete reader in shifted zone and create new one in healthy AZ.
• Monitor per-AZ CloudWatch metrics (healthy host count, CPU, memory, replica lag) throughout 48–72 hour drill for auditable evidence.
• Restore carefully: cancel EKS shift first, then load balancer shift, scale back gradually, verify even traffic distribution.

Multi-day AZ evacuation drills provide regulatory compliance evidence and operational confidence that multi-AZ architectures deliver genuine sustained resilience under N-1 capacity.