Entropy-Guided KV Caching for Efficient LLM Inference
Problem Statement
In long-context inference the KV cache grows linearly with sequence length and dominates memory and decoding latency, which limits scalability. Existing compression methods often use uniform per-layer budgets, which ignore how differently layers use context. Others select tokens separately per head, which complicates memory layout and can break the structural coherence of multi-head attention.
Key Novelty
- Uses the mean Shannon entropy of attention weights across a layer's heads as a measure of that layer's contextual importance, which drives KV budget allocation.
- Allocates cache budgets non-uniformly: more to high-entropy (broadly attending) layers, less to low-entropy (attention-sink-like) layers.
- Selects one common set of important tokens per layer from aggregated attention scores and shares it across all heads. This avoids per-head selection while keeping multi-head structure intact and keeping prefill-time selection efficient.
Evaluation Highlights
- Qwen3 4B: 4.18% reduction in memory usage with ROUGE score preserved.
- Mistral 7B (v0.1): 46.6% reduction in decoding time, with generation quality reportedly maintained.
Signal Assessment
Methodology
- During prefill, compute the attention weight distributions for each head in each Transformer layer and calculate their entropy.
- Average head entropies within each layer to get a layer-level contextual-importance score.
- Distribute the total KV cache budget across layers in proportion to the entropy scores, giving high-entropy layers more slots and low-entropy, sink-like layers fewer.
- Within each layer, aggregate attention scores across heads to rank tokens and pick the top-k under that layer's budget.
- Cache the selected shared token set uniformly for all heads in the layer, then run decoding with the compressed cache.
System Components
Computes Shannon entropy of each head's attention distribution to measure how dispersed or focused the attention is.
Averages head entropies within a layer into a single score for that layer's reliance on broad context.
Splits the global KV cache budget across layers by entropy, shrinking budgets for sink-dominated low-entropy layers.
Aggregates attention scores across heads to choose one common set of important tokens per layer, kept identically for all heads to preserve multi-head structure and simplify memory layout.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Memory usage (Qwen3 4B) | Baseline KV caching (not specified in abstract) | Reduced | -4.18% |
| ROUGE score (Qwen3 4B) | Baseline ROUGE (not specified in abstract) | Preserved | ~0 (no degradation reported) |
| Decoding time (Mistral 7B v0.1) | Baseline decoding latency (not specified in abstract) | Reduced | -46.6% |
| Generation quality (overall) | Baseline quality | Reported as not compromised | Qualitatively on par |
Key Takeaways
- Attention entropy, computed during prefill, is a cheap diagnostic for which layers can tolerate aggressive KV eviction. Sink-like low-entropy layers are good candidates for small budgets.
- Sharing one token set across all heads in a layer gives a simpler, more hardware-friendly cache layout than per-head selection. It may cost some fidelity on heads with specialized retrieval roles, so validate on needle-in-a-haystack-style tasks.
- The memory savings reported here are small and evaluation is ROUGE-centric. Before adopting this over PyramidKV, SnapKV or CAKE, benchmark against them on LongBench or RULER at matched budgets; the method can likely be combined with KV quantization for further gains.
Abstract
Large language models (LLMs), built upon Transformer architectures, have demonstrated remarkable performance in a wide range of natural language processing tasks. However, their practical deployment—especially in long-context scenarios—is often hindered by the computational and memory costs associated with managing the key–value (KV) cache during inference. Optimizing this process is therefore crucial for improving LLM efficiency and scalability. In this study, we propose a novel entropy-guided KV caching strategy that leverages the distribution characteristics of attention scores within each Transformer layer. Specifically, we compute the entropy of attention weights for each head and use the average entropy of all heads within a layer to assess the layer’s contextual importance. Higher-entropy layers—those exhibiting broader attention dispersion—are allocated larger KV cache budgets, while lower-entropy (sink-like) layers are assigned smaller budgets. Instead of selecting different key–value tokens per head, our method selects a common set of important tokens per layer, based on aggregated attention scores, and caches them uniformly across all heads within the same layer. This design preserves the structural integrity of multi-head attention while enabling efficient token selection during the prefilling phase. The experimental results demonstrate that our approach improves cache utilization and inference speed without compromising generation quality. For example, on the Qwen3 4B model, our method reduces memory usage by 4.18% while preserving ROUGE score, and on Mistral 0.1v 7B, it reduces decoding time by 46.6%, highlighting entropy-guided layer analysis as a principled mechanism for scalable long-context language modeling.