Accelerate inference with KV cache tiering on AWS

This article presents a four-tier KV cache storage framework for optimizing large language model inference at scale on AWS, addressing the challenge that KV cache becomes the binding constraint on accelerator utilization and cost per token.

• KV cache footprint grows linearly with context length × concurrent users × model layers, potentially reaching petabytes at scale
• Four-tier hierarchy: T1 (HBM, sub-microsecond), T2 (DRAM, single-digit microseconds), T3 (FSx for Lustre, 100-500 microseconds), T4 (S3, milliseconds to seconds)
• T3 enables fleet-wide prefix cache sharing, cross-node KV availability, and load balancing across inference clusters
• KV cache sizing formula: 2 × num_layers × num_kv_heads × head_dim × bytes_per_element; quantization reduces footprint by 50-75%
• Eviction policies (LRU, TTL-based) and promotion mechanisms determine whether cache hits feel instant or recomputation is preferable
• FSx for Lustre provides shared POSIX access, sub-millisecond latency, and hundreds of GBps throughput for disaggregated inference architectures
• Deploy T3 when KV working set exceeds fleet HBM, prefix sharing exceeds 30%, or adopting disaggregated prefill/decode pools

Proper KV cache tiering reduces redundant recomputation, lowers per-token costs, and enables disaggregated inference with independent prefill and decode scaling.