Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

This article presents three detection checks for silent data corruption (SDC) in distributed PyTorch training on Amazon SageMaker HyperPod, where hardware returns incorrect values without raising errors.

• Non-finite gradient check catches NaN, Inf, and overflow in forward/backward passes before all-reduce
• Ring replay audit recomputes micro-batches from other nodes to detect finite wrong values by comparing loss and gradient fingerprints
• Weight fingerprint comparison after optimizer step catches corruption in stored weights by comparing replicas
• Suspect rank maps to instance ID via Kubernetes node names for automatic node quarantine and replacement
• Tested on 4 ml.p5.48xlarge instances with 32 H100 GPUs; deterministic audits every 100 steps add ~10% overhead
• False-alarm rate bounded at ~1 in 5,100 audit steps based on 15,320 fault-free audits

The solution integrates detection in training code with SageMaker HyperPod's node replacement capabilities to automatically recover from silent data corruption without manual intervention.