Home icon

Amazon ElastiCache now supports distributed KV caching for self-hosted LLM inference

News



Amazon ElastiCache for Valkey now offers fully managed distributed KV caching to optimize self-hosted LLM inference performance and reduce costs.

  • Reduces inference costs by up to 40% and time to first token by up to 80%
  • Extends GPU-local caches with a shared tier for KV state reuse across sessions and servers
  • Runs on cache.i8ge.48xlarge nodes connected via RDMA over Elastic Fabric Adapter
  • Compatible with vLLM, SGLang, and TensorRT-LLM through LMCache connector
  • Works with open-weight, fine-tuned, and custom models with various attention designs
  • Available in 13 AWS Regions across US, Europe, and Asia Pacific
  • Use KVTrace tool to estimate cost and latency reductions before deployment

Distributed KV caching enables efficient GPU resource utilization by allowing servers to share cached KV state, reducing redundant computation and improving response times for LLM inference workloads.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 18
2026
Announcing distributed KV caching for self-hosted LLMs with Amazon ElastiCache
Nov 17
2026
Amazon DynamoDB caching is now available with Amazon ElastiCache Serverless for Valkey
Nov 17
2026
Amazon SageMaker AI now supports KV cache-aware routing for real-time inference endpoints
Sep 21
2026
Accelerate inference with KV cache tiering on AWS

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.