Amazon SageMaker HyperPod now supports on-demand deep health checks

This article announces on-demand deep health checks for Amazon SageMaker HyperPod clusters, enabling proactive GPU accelerator health verification on running instances.

• Supports Amazon EKS and Slurm-orchestrated clusters with comprehensive hardware stress tests
• Slurm clusters can run deep health checks during node provisioning at cluster creation
• Targets entire instance groups or specific instances for connectivity and GPU health verification
• Progress and results visible through SageMaker console and APIs at instance group and instance levels
• Instances undergoing checks automatically isolated from workload scheduling
• Failed instances automatically rebooted or replaced with automatic node recovery capability
• Available in all regions where SageMaker HyperPod is available

On-demand deep health checks help prevent compute resource waste by identifying unhealthy nodes before workload execution, improving cluster reliability and job performance.