Disaggregated prefill and decode for LLM inference on SageMaker HyperPod

This article explains Disaggregated Prefill and Decode (DPD), a technique for optimizing LLM inference on Amazon SageMaker HyperPod by running prefill and decode phases on separate GPU pools connected via Elastic Fabric Adapter (EFA) with Remote Direct Memory Access (RDMA).

• Prefill (compute-bound) and decode (memory-bound) phases compete for GPU resources when colocated, causing long prompts to stall token generation for concurrent requests.
• DPD removes interference by isolating each phase, enabling independent tuning of time-to-first-token (TTFT) and inter-token latency (ITL).
• Architecture uses vLLM Production Stack router, LMCache for KV cache transfer, and NIXL for unified memory abstraction across GPU, CPU, and remote peers.
• Ideal for long-context (4,096+ tokens), high-concurrency streaming workloads like chat assistants, agentic pipelines, and RAG systems.
• Benchmarks show 22-66% TPOT improvement on H100 and 28-48% on H200 at concurrency 8-32; throughput improves up to 35-64% at higher concurrency.
• Deployment uses HyperPod Inference Operator v3.2+ with P5/P6 instance families supporting NVLink and EFA RDMA in same Availability Zone.
• Short prompts bypass prefiller via configurable token threshold (default 4,096), avoiding unnecessary KV transfer overhead.

DPD enables predictable latency under mixed traffic and independent scaling of prefill and decode phases, making it ideal for production LLM serving at scale.