Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod
Machine Learning Blog
This article announces new resiliency features for Amazon SageMaker HyperPod that detect training failures beyond hardware diagnostics, including workload recovery and infrastructure monitoring capabilities.
- MultiNodeTrainingValidation deep health check trains a reference workload on instances to catch GPUs producing NaN/infinite loss despite passing hardware tests
- HyperPod Kubernetes Resource Monitoring Agent watches cluster infrastructure for pod crashes, missing DaemonSets, node taints, and NCCL communication issues
- Hung job detection identifies Slurm jobs that stop making progress while remaining active, with optional automatic cancellation and requeuing
- Available on EKS clusters (first two features) and Slurm clusters (hung job detection) with November 2026+ AMIs
- Reduces wasted GPU time by detecting failures in minutes rather than hours, saving thousands in compute costs
These capabilities improve training reliability by catching numerical faults, cluster-level failures, and stalled workloads that node-level health checks miss, enabling faster failure detection and recovery.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2026
2026
2025
2026
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.