Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference

Amazon SageMaker Inference introduces prefix-aware routing, a new strategy that sends requests sharing the same prompt prefix to the same instance to maximize KV cache reuse and reduce latency.

• Routes requests with identical prompt beginnings to the same instance for consistent cache hits
• Reduces P50 time-to-first-token by up to 77% and increases throughput by up to 16% on Llama 3.1 70B
• Improves KV cache hit rates from ~25% to over 80% in long-context workloads
• Includes overload protection and stable scaling behavior to prevent cache invalidation
• Configured via PrefixLength and ConcurrencyThreshold parameters in endpoint configuration
• Works with vLLM, TensorRT-LLM, inference components, and LoRA adapters
• Ideal for RAG applications, multi-turn conversations, templated bots, and code completion

Prefix-aware routing enables significant performance gains for LLM workloads by ensuring shared prompt context is cached efficiently across distributed inference endpoints.