Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
Problem Statement
LLM inference serving is bottlenecked by the memory footprint and management overhead of the KV cache, which grows with sequence length and batch size and directly limits latency and throughput. Prior optimization techniques (scheduling, memory placement, compression, eviction, etc.) have proliferated rapidly but in a fragmented way, making it hard for researchers and practitioners to see unifying principles, trade-offs, or under-explored combinations across techniques.
Key Novelty
- A new three-dimensional taxonomy (temporal execution/scheduling, spatial placement/migration, structural representation/retention) that reframes KV cache optimization from a system-behavior perspective rather than a purely algorithmic one
- Cross-behavior co-design analysis identifying affinities and synergies between techniques operating in different dimensions (e.g., scheduling combined with compression or migration)
- Behavior-objective mapping that explicitly links specific system behaviors/techniques to serving objectives such as latency, throughput, and memory efficiency
- Consolidation of a rapidly evolving, scattered research area into a coherent conceptual foundation intended to guide future KV cache infrastructure design
Evaluation Highlights
- No empirical benchmarks are reported; this is a survey whose contribution is validated through comprehensive literature synthesis and taxonomic organization rather than experiments
- Coverage and structure of the taxonomy (three orthogonal system-behavior dimensions plus cross-dimensional analysis) serve as the paper's primary form of 'evaluation'
Signal Assessment
Methodology
- Collect and review recent literature on KV cache management techniques across LLM serving systems
- Classify techniques according to the system behavior they primarily affect: temporal (execution/scheduling), spatial (placement/migration), or structural (representation/retention)
- Analyze cross-dimensional interactions to identify co-design affinities where techniques from different dimensions can be combined effectively
- Map identified behaviors and techniques to concrete serving objectives (e.g., latency, throughput, memory footprint) to reveal gaps and future research directions
System Components
Techniques governing when KV cache operations occur, including request/batch scheduling, prefill-decode coordination, and eviction timing to optimize latency and throughput
Techniques governing where KV cache resides across the memory/storage hierarchy (GPU, CPU, disk, remote) and how cache data is moved or offloaded between tiers or devices
Techniques governing how KV cache entries are encoded, compressed, quantized, or selectively retained/evicted to reduce memory footprint while preserving model quality
A framework for examining how techniques from the three dimensions can be jointly applied, revealing synergies and potential conflicts
An analytical mapping connecting specific system behaviors/techniques to the serving-level objectives they optimize (latency, throughput, memory efficiency, accuracy retention)
Results
| Aspect | Prior Landscape | This Survey's Contribution | Improvement |
|---|---|---|---|
| Organization of KV cache techniques | Fragmented, method-specific descriptions across many papers | Unified three-dimensional taxonomy based on system behavior | Improved conceptual clarity and comparability across techniques |
| Cross-technique relationships | Largely unexamined or implicit | Explicit cross-behavior co-design affinity analysis | Reveals combinable optimization strategies |
| Technique-to-goal mapping | Ad hoc, scattered across individual works | Explicit behavior-objective links (latency/throughput/memory) | Clearer guidance on selecting techniques for deployment goals |
Key Takeaways
- Practitioners can use the temporal/spatial/structural framework as a checklist when designing or auditing KV cache management strategies for a serving system
- Combining techniques across dimensions (e.g., pairing a scheduling policy with a compression or placement strategy) may yield larger efficiency gains than optimizing a single dimension in isolation
- Selecting KV cache optimizations should be driven by the target serving objective (latency-critical vs. throughput-critical vs. memory-constrained deployments), since different techniques align with different goals
- This survey serves as a reference map for identifying under-explored combinations and gaps in current KV cache research, useful for prioritizing future system design work
Abstract
Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling low-latency, high-throughput LLM inference serving. In this survey, we focus on system-aware KV infrastructure for serving LLMs (abbreviated as sKis). We revisit recent work from a system behavior perspective, organizing existing efforts into three dimensions: execution and scheduling (temporal), placement and migration (spatial), and representation and retention (structural). Furthermore, we analyze cross-behavior co-design affinity and behavior-objective links, highlighting future opportunities. Our work systematizes a rapidly evolving area, providing a foundation for understanding and innovating KV cache designs in modern LLM serving infrastructure.