← Back to Paper Radar

LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, Junchen Jiang
arXiv.org | 2025
LMCache is an open-source KV cache layer that moves KV caches out of GPU memory into CPU, storage and network tiers so they can be reused across queries and shared across inference engines (vLLM, SGLang). Optimized data movement and a modular engine connector make cross-query prefix reuse and prefill-decode disaggregation practical at enterprise scale.

Problem Statement

Real-world usage shows that the KV cache users store grows far beyond GPU memory capacity, so GPU-resident caches waste reusable computation across queries and engines. There was no efficient, general open-source solution for offloading and transferring KV caches. Building one is hard because naive data movement is I/O-bound and inference engines change their internals quickly, which breaks tight integrations.

Key Novelty

  • Highly optimized KV cache data movement using batched transfer operations and compute/I/O pipelining to hide offload and fetch latency
  • A modular KV cache connector that decouples the cache layer from fast-changing inference engine internals, supporting both vLLM and SGLang
  • A first-class control API for orchestrating KV caches across GPU, CPU, storage and network layers, covering both prefix offloading and PD-disaggregated cross-engine transfer
  • Empirical insights from enterprise deployment, e.g., widely used context truncation can cut prefix cache hit ratio by about half

Evaluation Highlights

  • vLLM combined with LMCache achieves up to 15x throughput improvement on workloads such as multi-round QA and document analysis
  • Production observations: fetching KV cache from remote storage reduces prefill delay, and context truncation can roughly halve the prefix cache hit ratio

Signal Assessment

6/10 LMCache is a strong, widely adopted systems contribution that makes KV cache offloading and PD disaggregation practical, with large throughput gains. Its core ideas (tiered KV storage, prefix reuse, pipelined I/O) build on known concepts, so it is engineering-heavy rather than a paradigm shift.

Methodology

  1. Extract KV caches produced by vLLM/SGLang during prefill through a modular connector interface that hides engine-specific internals
  2. Move KV chunks between GPU and external tiers (CPU memory, local/remote storage, network peers) using batched operations, pipelined with computation to overlap I/O
  3. Index stored KV caches so future queries with matching prefixes, or decode engines in PD disaggregation, can retrieve them instead of recomputing prefill
  4. Expose a control API that lets operators or schedulers manage cache placement, eviction and transfer across tiers and engines
  5. Evaluate throughput and latency on multi-round QA and document analysis workloads, and analyze real enterprise usage traces

System Components

KV Cache Connector

Modular adapter between inference engines (vLLM, SGLang) and LMCache that isolates the cache layer from engine API and memory-layout changes

Optimized Data Movement Engine

Batched GPU↔CPU/storage/network transfers with compute–I/O pipelining to minimize offload and load overhead

Multi-tier Storage Backend

Hierarchical KV storage spanning GPU, CPU DRAM, local disk/SSD and remote storage to hold caches far exceeding GPU capacity

Control/Orchestration API

First-class interface for managing cache placement, lookup, eviction and cross-engine transfer policies

PD Disaggregation Transfer Path

Moves KV caches from prefill engines/GPUs to decode engines/GPUs over the network for disaggregated serving

Results

Metric/Benchmark Baseline This Paper Delta
Throughput (multi-round QA, document analysis) vLLM without LMCache vLLM + LMCache Up to 15x higher
Prefill delay with remote KV fetch Recompute prefill Fetch KV from remote storage Reduced (qualitative)
Prefix cache hit ratio under context truncation No truncation With truncation Roughly halved (~50% drop)
Engine support Engine-specific/ad hoc caching vLLM and SGLang via modular connector Cross-engine portability

Key Takeaways

  • For multi-turn chat, agentic loops and repeated long-document workloads, offloading KV caches to CPU or storage and reusing prefixes can yield large throughput gains. LMCache is a drop-in open-source option for vLLM/SGLang.
  • Context truncation can roughly halve prefix cache hit rates. Agent and RAG pipelines should keep prompt prefixes stable, for example by truncating from the middle or keeping a fixed system/context prefix.
  • Loading KV from remote storage is often faster than recomputing prefill, so tiered KV storage and PD disaggregation are worth adopting for long-context serving, provided I/O is batched and pipelined.
  • Decoupling cache infrastructure from engine internals through a connector abstraction is a practical design pattern given how quickly inference engines evolve.

Abstract

KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV caches outside GPU devices, to enable cache reuse across different queries and inference engines. Our real-world usage statistics confirm this trend: over time, the total KV cache stored by users has grown rapidly, far exceeding the capacity of GPU memory. Despite this need, there lacks an efficient solution for offloading and transferring KV caches. We present LMCACHE, the first and so far the most efficient open-source KV caching solution, which extracts and stores KV caches generated by modern LLM engines (vLLM and SGLang) out of the GPU memory and shares them across engines and queries. LMCACHE supports both cache offloading (prefix reuse across queries) and prefill-decode (PD) disaggregation (cross-engine/GPU cache transfer). LMCACHE's high performance and wide adoption stem from the following contributions: (1) highly optimized KV cache data movement powered by batched data movement operations, compute and I/O pipelining; (2) a modular KV cache connector component, decoupling LMCACHE from the rapid evolution of inference engines; (3) a first-class control API for flexible cache orchestration across GPU, CPU, storage, and network layers. Our evaluation shows that combining LMCACHE with vLLM achieves up to 15x improvement in throughput across workloads such as multi-round question answering and document analysis. Large-scale adoption of LMCACHE in enterprise settings provides us valuable insights, for example, fetching KV cache from remote storage has unsurprisingly benefits to prefill delay, and that context truncation, which is a widely applied technique in industry, can greatly reduce prefix cache hit ratio by half. The source code of LMCACHE is at: https://github.com/LMCache/LMCache.

Generated automatically from available metadata and abstract on 2026-11-09.