VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
Problem Statement
KV cache memory grows linearly with sequence length and batch size, which makes it a dominant bottleneck for long-context and large-batch LLM serving. Vector quantization offers flexible bit-width trade-offs, but it degrades badly at ultra-low bits. The cause is channel-wise outliers in the key cache, which distort codebook learning and leave codewords poorly utilized. Existing methods therefore struggle to get both aggressive compression and near-lossless accuracy with practical speed gains.
Key Novelty
- Diagnoses key-cache outliers as the root cause of VQ failure at ultra-low bit-widths, through poor codebook coverage and utilization.
- Combines smooth (channel-wise scaling) and Hadamard rotation transformations to make the key distribution more uniform and easier to quantize before VQ, adapting outlier-suppression ideas from scalar quantization (SmoothQuant, QuaRot-style) to vector quantization of the KV cache.
- Introduces an optimized CUDA kernel that fuses codebook dequantization with attention computation, avoiding materialization of full-precision KV and minimizing memory traffic.
Evaluation Highlights
- At 2-bit KV quantization, accuracy is comparable to full precision on long-context understanding and mathematical reasoning tasks, and consistently above existing quantization baselines.
- Up to 2.7× speedup in large-batch self-attention computation and 8.3× reduction in single-batch end-to-end latency on Llama-3.1-8B at 196k sequence length.
Signal Assessment
Methodology
- Analyze the KV cache distribution and identify channel-wise outliers in keys that hinder codebook coverage under vector quantization.
- Apply a smooth transformation (per-channel scaling) to redistribute outlier magnitude, followed by a Hadamard rotation to further spread energy across dimensions and produce a more isotropic key distribution.
- Learn VQ codebooks on the transformed key and value vectors and quantize KV entries into compact codeword indices, targeting about 2 bits per element.
- Fold or invert the transformations consistently in the attention computation so outputs are preserved; for example, the orthogonal Hadamard rotation can be applied to queries as well.
- Implement a fused CUDA kernel that performs codebook lookup and dequantization inside the attention computation to reduce memory bandwidth.
- Evaluate on long-context understanding and mathematical reasoning benchmarks, and profile attention-kernel and end-to-end latency at long sequence lengths and varied batch sizes.
System Components
Per-channel scaling that dampens large-magnitude outlier channels in the key cache, reducing dynamic range disparities before quantization.
Orthogonal rotation that spreads outlier energy across dimensions, yielding a more uniform distribution that VQ codebooks can cover effectively.
Learned codebooks that map groups of KV elements to codeword indices, enabling flexible ultra-low bit-width (e.g., 2-bit) representation.
Custom kernel that combines codebook lookup and dequantization with attention computation to minimize memory access and deliver wall-clock speedups.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Task accuracy at 2-bit KV (long-context and math reasoning) | Existing VQ/scalar KV quantization baselines degrade at ultra-low bits | Comparable to full precision (FP16) | Consistently outperforms baselines; near-lossless vs. FP16 |
| Large-batch self-attention speed | Full-precision attention | Up to 2.7× faster | +2.7× speedup |
| Single-batch end-to-end latency (Llama-3.1-8B, 196k context) | Full-precision inference | Up to 8.3× lower latency | 8.3× reduction |
| KV cache memory footprint | 16-bit KV cache | About 2-bit KV cache | About 8× compression (nominal) |
Key Takeaways
- Outlier suppression via smoothing and Hadamard rotation is not just for scalar quantization. It substantially improves vector quantization of the KV cache and makes 2-bit KV practical for long-context workloads.
- Memory compression only yields real speedups with fused kernels. Dequantizing inside the attention kernel is essential to convert bandwidth savings into latency gains, especially at very long contexts.
- For long-context serving (100k+ tokens) or large-batch deployments, a 2-bit VQ KV cache can raise achievable context length and batch size with little accuracy loss, including on reasoning-heavy tasks like math.
Abstract
The Key-Value (KV) cache introduces substantial memory overhead during large language model (LLM) inference. Although existing vector quantization (VQ) methods reduce KV cache usage and provide flexible representational capacity across bit-widths, they suffer severe performance degradation at ultra-low bit-widths due to key cache outliers that hinder effective codebook utilization. To address this challenge, we propose VecInfer, a novel VQ method for aggressive KV cache compression while enabling efficient inference. By applying smooth and Hadamard transformations, VecInfer suppresses outliers in the key cache, enabling the codebook to comprehensively cover the original data distribution and thereby reducing quantization difficulty. To facilitate efficient deployment, we design an optimized CUDA kernel that fuses computation with dequantization to minimize memory access overhead. Extensive evaluations demonstrate that VecInfer consistently outperforms existing quantization baselines across both long-context understanding and mathematical reasoning tasks. With only 2-bit quantization, VecInfer achieves performance comparable to full precision, while delivering up to $\mathbf{2.7\times}$ speedup in large-batch self-attention computation and $\mathbf{8.3\times}$ reduction in single-batch end-to-end latency on Llama-3.1-8B with a 196k sequence length.