This article announces Managed Tiered KV Cache and Intelligent Routing capabilities for Amazon SageMaker HyperPod, designed to optimize LLM inference performance and reduce costs.

• Reduces time-to-first-token (TTFT) by up to 40% for long context workloads
• Increases throughput by up to 38% and reduces compute costs by up to 28%
• Two-tier KV cache: L1 (local CPU memory) and L2 (distributed cluster-wide storage)
• L2 cache supports AWS-managed tiered storage or Redis backends
• Four intelligent routing strategies: prefix-aware, KV-aware, round-robin, and default
• Automatic cache management eliminates manual configuration overhead
• Particularly beneficial for long documents, multi-turn conversations, and high-throughput inference
• Built-in observability integration with Amazon Managed Grafana
• Available in all AWS regions where SageMaker HyperPod is supported

These features enable enterprise-scale LLM deployments with significantly improved performance and cost efficiency, especially for applications processing long contexts or maintaining multi-turn conversations.