Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM

This article demonstrates deploying Qwen3.8-2.4T-A95B, a 2.4-trillion-parameter open-weight model, on Amazon SageMaker HyperPod using vLLM with NVFP4 quantization on a single ml.p6-b300 instance.

• Qwen3.8 features 95B activated parameters, hybrid linear-plus-full attention, 262K native context window, and built-in reasoning controls
• NVFP4 quantization compresses the 2.4T model to ~1.2TB, fitting within 2.1TB GPU memory on 8× NVIDIA B300 Blackwell Ultra GPUs
• SageMaker HyperPod provides EKS orchestration, automatic model download, health monitoring, and resilience for large-scale inference
• vLLM configuration enables prefix caching, native Multi-Token Prediction speculative decoding, tool calling, and reasoning output parsing
• Benchmarks show 59.7% TTFT reduction and 12.6% throughput increase with Expert Parallelism and MTP combined
• OpenAI-compatible API supports streaming, tool calling with strict schema validation, and per-request reasoning depth control

The deployment provides a production-ready endpoint for agentic workloads including multi-step coding, long-horizon planning, and autonomous tool use with minimal operational overhead.