Amazon SageMaker HyperPod now supports deep health checks for Slurm clusters with continuous provisioning

This article announces deep health checks for Amazon SageMaker HyperPod Slurm-orchestrated clusters with continuous provisioning, enabling proactive GPU accelerator health validation.

• Run comprehensive hardware stress tests and connectivity tests on instance groups or specific instances before job scheduling
• Continuous provisioning adds worker nodes asynchronously without all-or-nothing failures
• Deep health checks validate hardware on new nodes as they come online without interrupting running workloads
• Progress and results visible at instance group and instance level through SageMaker console and APIs
• Instances undergoing checks are automatically isolated from workload scheduling
• Failed instances automatically rebooted or replaced via HyperPod's automatic node recovery
• Available in all regions where Amazon SageMaker HyperPod is available

Deep health checks prevent wasted compute time by ensuring cluster health before committing resources to jobs.