Announcing distributed KV caching for self-hosted LLMs with Amazon ElastiCache
Database Blog
This article announces general availability of distributed KV caching with Amazon ElastiCache for Valkey, a capability that reduces self-hosted LLM inference costs by up to 40% and time to first token (TTFT) by up to 80%.
- Distributed KV caching adds a shared L2 cache tier for GPU fleets to store key-value state computed during prefill phase
- Connects to existing inference engines (vLLM, SGLang, TensorRT-LLM) via open-source LMCache connector with configuration changes only
- Uses RDMA over EFA for sub-millisecond latency and up to 220 Gbps throughput between GPU nodes and cache nodes
- Supports data tiering with hot KV blocks in DRAM and cold blocks automatically moved to NVMe for cost-effective storage
- Tested across coding agents, AI assistants, and long-context Q&A workloads showing 23-100% throughput improvements and 55-93% TTFT reductions
- KVTrace tool available to estimate cost savings and TTFT improvements for specific workloads before deployment
Distributed KV caching enables GPU fleets to share computed work across servers, improving performance for long-running sessions, handling traffic spikes, and surviving server failures without recomputation.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2026
2026
2026
2026
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.