← Back to Paper Radar

VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization

Dingyu Yao, Chenxu Yang, Zhengyang Tong, Zheng Lin, Wei Liu, Jian Luan, Weiping Wang
arXiv.org | 2025
VecInfer makes vector quantization of the KV cache work at ultra-low bit-widths (about 2 bits). It first applies smooth scaling and Hadamard rotations to suppress outliers in the key cache, so the VQ codebooks can cover the data distribution well. A fused CUDA kernel then turns the memory savings into real inference speedups.

Problem Statement

KV cache memory grows linearly with sequence length and batch size, which makes it a dominant bottleneck for long-context and large-batch LLM serving. Vector quantization offers flexible bit-width trade-offs, but it degrades badly at ultra-low bits. The cause is channel-wise outliers in the key cache, which distort codebook learning and leave codewords poorly utilized. Existing methods therefore struggle to get both aggressive compression and near-lossless accuracy with practical speed gains.

Key Novelty

  • Diagnoses key-cache outliers as the root cause of VQ failure at ultra-low bit-widths, through poor codebook coverage and utilization.
  • Combines smooth (channel-wise scaling) and Hadamard rotation transformations to make the key distribution more uniform and easier to quantize before VQ, adapting outlier-suppression ideas from scalar quantization (SmoothQuant, QuaRot-style) to vector quantization of the KV cache.
  • Introduces an optimized CUDA kernel that fuses codebook dequantization with attention computation, avoiding materialization of full-precision KV and minimizing memory traffic.

Evaluation Highlights

  • At 2-bit KV quantization, accuracy is comparable to full precision on long-context understanding and mathematical reasoning tasks, and consistently above existing quantization baselines.
  • Up to 2.7× speedup in large-batch self-attention computation and 8.3× reduction in single-batch end-to-end latency on Llama-3.1-8B at 196k sequence length.

Signal Assessment

6/10 This is a solid, practically valuable systems-plus-algorithms contribution. It shows near-lossless 2-bit VQ for the KV cache and delivers real kernel-level speedups. However, the core techniques (smoothing, Hadamard rotation, fused dequantization kernels) are known from scalar quantization work, so the advance is mainly in combining them effectively for VQ rather than in a new paradigm.

Methodology

  1. Analyze the KV cache distribution and identify channel-wise outliers in keys that hinder codebook coverage under vector quantization.
  2. Apply a smooth transformation (per-channel scaling) to redistribute outlier magnitude, followed by a Hadamard rotation to further spread energy across dimensions and produce a more isotropic key distribution.
  3. Learn VQ codebooks on the transformed key and value vectors and quantize KV entries into compact codeword indices, targeting about 2 bits per element.
  4. Fold or invert the transformations consistently in the attention computation so outputs are preserved; for example, the orthogonal Hadamard rotation can be applied to queries as well.
  5. Implement a fused CUDA kernel that performs codebook lookup and dequantization inside the attention computation to reduce memory bandwidth.
  6. Evaluate on long-context understanding and mathematical reasoning benchmarks, and profile attention-kernel and end-to-end latency at long sequence lengths and varied batch sizes.

System Components

Smooth Transformation

Per-channel scaling that dampens large-magnitude outlier channels in the key cache, reducing dynamic range disparities before quantization.

Hadamard Transformation

Orthogonal rotation that spreads outlier energy across dimensions, yielding a more uniform distribution that VQ codebooks can cover effectively.

Vector Quantization Codebooks

Learned codebooks that map groups of KV elements to codeword indices, enabling flexible ultra-low bit-width (e.g., 2-bit) representation.

Fused Dequantization-Attention CUDA Kernel

Custom kernel that combines codebook lookup and dequantization with attention computation to minimize memory access and deliver wall-clock speedups.

Results

Metric/Benchmark Baseline This Paper Delta
Task accuracy at 2-bit KV (long-context and math reasoning) Existing VQ/scalar KV quantization baselines degrade at ultra-low bits Comparable to full precision (FP16) Consistently outperforms baselines; near-lossless vs. FP16
Large-batch self-attention speed Full-precision attention Up to 2.7× faster +2.7× speedup
Single-batch end-to-end latency (Llama-3.1-8B, 196k context) Full-precision inference Up to 8.3× lower latency 8.3× reduction
KV cache memory footprint 16-bit KV cache About 2-bit KV cache About 8× compression (nominal)

Key Takeaways

  • Outlier suppression via smoothing and Hadamard rotation is not just for scalar quantization. It substantially improves vector quantization of the KV cache and makes 2-bit KV practical for long-context workloads.
  • Memory compression only yields real speedups with fused kernels. Dequantizing inside the attention kernel is essential to convert bandwidth savings into latency gains, especially at very long contexts.
  • For long-context serving (100k+ tokens) or large-batch deployments, a 2-bit VQ KV cache can raise achievable context length and batch size with little accuracy loss, including on reasoning-heavy tasks like math.

Abstract

The Key-Value (KV) cache introduces substantial memory overhead during large language model (LLM) inference. Although existing vector quantization (VQ) methods reduce KV cache usage and provide flexible representational capacity across bit-widths, they suffer severe performance degradation at ultra-low bit-widths due to key cache outliers that hinder effective codebook utilization. To address this challenge, we propose VecInfer, a novel VQ method for aggressive KV cache compression while enabling efficient inference. By applying smooth and Hadamard transformations, VecInfer suppresses outliers in the key cache, enabling the codebook to comprehensively cover the original data distribution and thereby reducing quantization difficulty. To facilitate efficient deployment, we design an optimized CUDA kernel that fuses computation with dequantization to minimize memory access overhead. Extensive evaluations demonstrate that VecInfer consistently outperforms existing quantization baselines across both long-context understanding and mathematical reasoning tasks. With only 2-bit quantization, VecInfer achieves performance comparable to full precision, while delivering up to $\mathbf{2.7\times}$ speedup in large-batch self-attention computation and $\mathbf{8.3\times}$ reduction in single-batch end-to-end latency on Llama-3.1-8B with a 196k sequence length.

Generated automatically from available metadata and abstract on 2026-11-09.