Amazon SageMaker HyperPod now supports checkpointless training

This article announces checkpointless training support in Amazon SageMaker HyperPod, a new capability for fault-tolerant foundational model training.

• Reduces failure recovery time from hours to minutes without checkpoint-based restarts
• Preserves model training state across distributed clusters automatically
• Swaps faulty nodes on-the-fly using peer-to-peer state transfer from healthy accelerators
• Achieves 95% training goodput even on clusters with thousands of AI accelerators
• Eliminates expensive AI accelerator idle time and associated compute costs
• Available in all AWS Regions where SageMaker HyperPod operates
• Zero code changes needed for popular models like Llama and GPT OSS
• Minimal modifications required for custom PyTorch-based workflows

Checkpointless training fundamentally improves fault recovery efficiency in large-scale AI model training, significantly reducing costs and accelerating training timelines.