Home icon

Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod

Machine Learning Blog



This article announces enhanced observability capabilities in Amazon SageMaker HyperPod to help diagnose GPU training issues faster across distributed training workloads.

  • Diagnostic summary uses CloudWatch DetectGroupOutliers API to flag GPUs deviating from peers in utilization, memory, temperature, power, and clock speed
  • Static diagnostic rules surface common failure patterns including GPU hardware faults, EFA throughput drops, thermal throttling, and overheating
  • Default metrics (cluster health, CPU/disk/OS, GPU utilization/temperature) collected every 30 seconds with ~1-2% CPU overhead
  • Advanced metrics available for deeper visibility: full node exporter suite, NVIDIA GPU DCGM/Neuron metrics, and EFA network metrics
  • Supports both Amazon CloudWatch (new default) and Amazon Managed Service for Prometheus as metric stores
  • Available as EKS add-on or Slurm cluster extension with IAM-controlled installation
  • CloudWatch dashboard provides diagnostic summary, hardware fleet topology, workload monitoring, and network health views

Enhanced observability reduces troubleshooting time for expensive GPU training jobs by isolating performance issues and providing comprehensive visibility across infrastructure layers.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Jul 10
2025
Amazon SageMaker HyperPod announces new observability capability
Nov 18
2026
Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod
Nov 20
2026
Introducing managed cluster extensions for observability and user management in Amazon SageMaker HyperPod Slurm
Mar 4
2026
Amazon SageMaker HyperPod now provides comprehensive observability for Restricted Instance Groups

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.