Automatically recover from GPU failures with Amazon ECS Managed Instances

This article explains how Amazon ECS Managed Instances automatically recovers from GPU failures using NVIDIA DCGM integration and built-in health monitoring.

• ECS detects critical GPU hardware failures (double-bit ECC errors, NVLink failures, GPU bus disconnects) and marks instances IMPAIRED
• Auto-repair uses start-before-stop workflow: provisions replacement instance, drains impaired instance, terminates it after tasks stop
• Distinguishes critical hardware failures requiring replacement from application-level GPU faults surfaced through CloudWatch metrics
• Publishes health change events to Amazon EventBridge for custom alerting and automation workflows
• Includes fault injection testing with DCGM to validate auto-repair without affecting GPU hardware
• Sample dual-tier inference pipeline demonstrates recovery with Mistral-7B and Llama-2-70B models on ECS Managed Instances

GPU auto-repair eliminates manual remediation workflows, reducing operational complexity and recovery time for GPU inference workloads at scale.