Fault tolerant distributed training on Amazon EKS using NVRx

This article demonstrates how to integrate NVIDIA Resiliency Extension (NVRx) with PyTorch FSDP training on Amazon EKS to achieve fault-tolerant distributed training with async checkpointing and fast recovery.

• Async checkpointing overlaps I/O with training, achieving 99%+ efficiency versus 57-61% for synchronous checkpointing
• In-process restart recovers from soft faults (exceptions, NCCL hangs) in ~10 seconds without container restarts
• ft_launcher handles hard faults (SIGKILL, OOM) by respawning workers and reloading from latest checkpoint
• Benchmarks on H100 GPUs show NVRx in-process restart achieves 31% training goodput versus 11.5% for baseline Kubernetes recovery
• Solution uses Amazon EKS with p5.48xlarge nodes, EFA networking, and FSx for Lustre shared storage

NVRx enables aggressive checkpointing and rapid fault recovery, maximizing GPU utilization for large-scale distributed training workloads.