Home icon

Announcing distributed KV caching for self-hosted LLMs with Amazon ElastiCache

Database Blog



This article announces general availability of distributed KV caching with Amazon ElastiCache for Valkey, a capability that reduces self-hosted LLM inference costs by up to 40% and time to first token (TTFT) by up to 80%.

  • Distributed KV caching adds a shared L2 cache tier for GPU fleets to store key-value state computed during prefill phase
  • Connects to existing inference engines (vLLM, SGLang, TensorRT-LLM) via open-source LMCache connector with configuration changes only
  • Uses RDMA over EFA for sub-millisecond latency and up to 220 Gbps throughput between GPU nodes and cache nodes
  • Supports data tiering with hot KV blocks in DRAM and cold blocks automatically moved to NVMe for cost-effective storage
  • Tested across coding agents, AI assistants, and long-context Q&A workloads showing 23-100% throughput improvements and 55-93% TTFT reductions
  • KVTrace tool available to estimate cost savings and TTFT improvements for specific workloads before deployment

Distributed KV caching enables GPU fleets to share computed work across servers, improving performance for long-running sessions, handling traffic spikes, and surviving server failures without recomputation.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 17
2026
Amazon DynamoDB caching is now available with Amazon ElastiCache Serverless for Valkey
Nov 12
2026
Building a self-service LLM gateway with Amazon Bedrock and LiteLLM
Nov 17
2026
Accelerate your DynamoDB workloads with DynamoDB caching with Amazon ElastiCache Serverless for Valkey
Nov 11
2026
How Flickr modernized caching with Amazon ElastiCache for Valkey

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.