Building resilient real-time streaming workers with Amazon DynamoDB leases

This article describes a DynamoDB-based lease pattern for managing distributed WebSocket connections across a fleet of ECS workers with automatic failover and graceful scaling.

• Use DynamoDB conditional writes to implement distributed leases that track connection ownership with automatic expiration
• Workers renew leases via heartbeats; lease expiration triggers orphan reconciliation to reassign connections to healthy workers
• Graceful shutdown releases leases immediately; unexpected terminations rely on lease expiration and reconciliation for recovery
• SQS provides fast notification of new connections; workers poll for work and acquire leases before opening WebSocket connections
• Three concurrent tasks manage each connection: heartbeat renewal, message receiving, and desired state checking
• Reconciliation loop queries a GSI for expired leases and reacquires orphaned connections within 60 seconds
• Fleet scales automatically based on CloudWatch metrics; scale-in is safe because leases are released before task termination
• Heartbeat writes dominate DynamoDB costs; provisioned capacity with auto-scaling is recommended for sustained high connection counts

This pattern eliminates manual intervention during worker failures, reduces recovery time from minutes to seconds, and enables safe horizontal scaling without external coordination services.