Home icon

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

Machine Learning Blog



This article announces new resiliency features for Amazon SageMaker HyperPod that detect training failures beyond hardware diagnostics, including workload recovery and infrastructure monitoring capabilities.

  • MultiNodeTrainingValidation deep health check trains a reference workload on instances to catch GPUs producing NaN/infinite loss despite passing hardware tests
  • HyperPod Kubernetes Resource Monitoring Agent watches cluster infrastructure for pod crashes, missing DaemonSets, node taints, and NCCL communication issues
  • Hung job detection identifies Slurm jobs that stop making progress while remaining active, with optional automatic cancellation and requeuing
  • Available on EKS clusters (first two features) and Slurm clusters (hung job detection) with November 2026+ AMIs
  • Reduces wasted GPU time by detecting failures in minutes rather than hours, saving thousands in compute costs

These capabilities improve training reliability by catching numerical faults, cluster-level failures, and stalled workloads that node-level health checks miss, enabling faster failure detection and recovery.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 17
2026
Amazon SageMaker HyperPod now supports cluster operations for tracking scaling and on-demand Deep Health Check requests
Nov 18
2026
Amazon SageMaker HyperPod now supports higher pod density on EKS-orchestrated clusters
Sep 8
2025
Announcing Managed Tiered Checkpointing for Amazon SageMaker HyperPod
Nov 11
2026
Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.