Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine

This article describes a tiered KV cache architecture on Amazon SageMaker HyperPod that extends GPU and CPU memory into a shared distributed NVMe pool using Curvine, enabling efficient LLM inference on cost-effective instances.

• Three-tier cache hierarchy: L0 (GPU HBM), L1 (CPU memory offload via LMCache), L2 (shared Curvine distributed NVMe pool)
• Intelligent routing directs requests to replicas with highest cache hit probability using prefix-aware or KV-aware strategies
• Achieved up to 2.7x TTFT improvement and 100% cross-Pod cache hit rates on long prompts (1,900+ tokens)
• Cross-node L2 read latency approximately 56 ms; writes to node-local NVMe at 9.6 GB/s
• Enables large models on cost-efficient ml.g6e.4xlarge instances instead of expensive P5 or larger G6e variants
• Step-by-step implementation covers enabling HyperPod Tiered Storage, deploying Curvine, and patching the Inference Operator

This architecture suits workloads with high prompt overlap like RAG pipelines and multi-turn dialogue, reducing infrastructure costs while maintaining low latency for LLM inference at scale.