Amazon SageMaker AI now supports KV cache-aware routing for real-time inference endpoints
News
This article announces KV cache-aware routing support in Amazon SageMaker AI for real-time inference endpoints, reducing time to first token for LLM workloads with long or repeated prompts.
- Routes requests to instances holding the longest matching prompt cache to avoid recomputation
- vLLM container reports cache blocks stored and evicted for live routing decisions
- Falls back to load-aware routing if cache lookup is slow or finds no match
- Enable by setting RoutingStrategy to KV_AWARE and adding one environment variable
- Supports single-model endpoints, inference components, and LoRA adapters on GPU instances
- Requires AWS vLLM Deep Learning Container version 0.33.0 or later with at least two instances
- Provides observability metrics showing KV cache hit rate with no additional charge
- Available in all AWS Regions where SageMaker AI is available
KV cache-aware routing optimizes LLM inference performance for long prompts and high-load scenarios by intelligently routing requests to instances with relevant cached data.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Oct 30
2026
2026
Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints
Nov 27
2025
2025
Managed Tiered KV Cache and Intelligent Routing for Amazon SageMaker HyperPod
May 21
2026
2026
Amazon SageMaker AI now supports OpenAI-compatible APIs for inference endpoints
Jun 17
2026
2026
Amazon SageMaker AI Async Inference now supports inline request payloads
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.