Run multi-node inference workloads using Ray LLM Deep Learning Containers

This article demonstrates how to deploy a large language model across two GPU nodes on Amazon EKS using Ray LLM Deep Learning Containers with prefill/decode disaggregation.

• Separate prefill and decode inference phases onto different GPU nodes to prevent resource contention and improve performance
• Ray LLM DLC bundles vLLM, Ray Serve, NIXL, and EFA stack for high-bandwidth node-to-node communication
• Deploy using YAML configuration with build_pd_openai_app helper; no custom handler code or model archiver required
• KubeRay operator orchestrates Ray cluster; NIXL transfers KV cache between nodes using libfabric backend
• Complete deployment includes CPU system node for Ray head and two g6.8xlarge GPU worker nodes in single availability zone
• OpenAI-compatible API endpoint enables standard chat-completion requests without custom schema

Multi-node Ray LLM DLC simplifies distributed LLM serving by providing pre-built, tested infrastructure stack for prefill/decode disaggregation and other parallelism patterns on Amazon EKS.