How Intuit and AWS systematically improved resiliency on ElastiCache using AWS Fault Injection Service

This article describes how Intuit and AWS systematically improved ElastiCache resiliency using AWS Fault Injection Service (FIS) to simulate Availability Zone failures and validate fixes at production-level load.

• Identified recurring cache-access disruptions traced to client misconfiguration, not ElastiCache itself
• Created a Resiliency Checklist covering client configuration, cluster architecture, observability, and operational readiness
• Simulated AZ failure using FIS action aws:elasticache:replicationgroup-interrupt-az-power to test failover behavior
• Validated Lettuce topology refresh, exponential backoff, and static stability enable automatic recovery without manual intervention
• Achieved 95% faster recovery time (under 2 minutes vs. 50+ minutes), 96% reduction in customer impact, and 93% reduction in operational load
• Published checklist as enterprise-wide reliability standard for all datastores across Intuit

By converting monitoring insights into tested controls and running production-scale GameDay exercises, Intuit transformed fragile caching infrastructure into resilient, self-healing systems.