Home icon

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

Machine Learning Blog



Amazon SageMaker AI introduces KV cache-aware routing, a new strategy for real-time endpoints that routes LLM requests to instances based on actual cached content state rather than request prefixes alone.

  • Routes requests to instances holding matching cached prompts, weighing cache match length against instance load
  • Solves cache eviction problems where prefix-aware routing keeps sending requests to instances that have dropped their cache
  • Supports multi-turn conversations, RAG applications, and workloads with shared long prompts under memory pressure
  • Requires AWS vLLM Deep Learning Container with KV cache events enabled via SM_VLLM_KV_EVENTS_CONFIG environment variable
  • Benchmarks show up to 97% reduction in time-to-first-token and 234% throughput increase versus random routing on long prompts
  • Enable by setting RoutingStrategy to KV_AWARE in endpoint configuration; no changes needed to client API calls
  • Available at no additional charge in all commercial AWS Regions supporting SageMaker real-time inference

KV-aware routing complements prefix-aware routing for LLM inference, automatically managing cache state without requiring fixed prefix length tuning.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 17
2026
Amazon SageMaker AI now supports KV cache-aware routing for real-time inference endpoints
Nov 27
2025
Managed Tiered KV Cache and Intelligent Routing for Amazon SageMaker HyperPod
Oct 30
2026
Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints
Nov 26
2025
SageMaker HyperPod now supports Managed tiered KV cache and intelligent routing

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.