Resilient HPC and ML on AWS: Running Tightly Coupled Workloads on Spot Instances

This article presents Smart Job Navigator, a solution that enables running tightly-coupled HPC and ML workloads on AWS EC2 Spot Instances with automatic interruption handling and intelligent queue management.

• Detects Spot interruptions via Slurm job state signals and automatically resubmits jobs to optimal queues using AWS Spot Placement Score API
• Supports checkpointing-enabled applications like STAR-CCM+, GROMACS, and CAE tools from Ansys, Siemens, and Dassault Systems
• Integrates with AWS ParallelCluster and AWS PCS through lightweight sjn-submit wrapper around Slurm sbatch command
• Delivers 50%+ cost savings with 14-27% wall-time overhead compared to On-Demand instances in real-world CFD and molecular dynamics tests
• Provides intelligent queue selection across multiple Availability Zones and instance types with configurable retry limits and comprehensive job tracking
• Requires applications with checkpointing capability and flexibility across at least 3 instance types for optimal Spot pool access

Smart Job Navigator bridges the cost advantages of Spot Instances with reliability requirements of long-running HPC jobs through automatic interruption handling, intelligent scheduling, and detailed performance metrics.