PipeRAG: Fast Retrieval-Augmented Generation via Adaptive Pipeline Parallelism

PipeRAG co-designs retrieval-augmented generation (RAG) algorithms with system execution by overlapping retrieval and generation through pipeline parallelism, using adaptive retrieval intervals and a performance model to jointly minimize latency and preserve/improve generation quality.

Problem Statement

In RAG systems that periodically re-retrieve from large token databases to keep context aligned with the model's evolving generation state, retrieval latency can dominate total inference time because retrieval and generation are typically executed sequentially. Existing systems treat the retrieval schedule and hardware execution as independent concerns, leaving substantial latency on the table as database size and retrieval frequency grow. This limits the practicality of high-quality, frequently-retrieving RAG systems in latency-sensitive deployment settings.

Key Novelty

Pipeline parallelism that overlaps retrieval computation with ongoing token generation instead of executing them sequentially
Flexible/adaptive retrieval intervals that adjust retrieval timing to maximize pipeline overlap efficiency rather than using fixed periodic schedules
A performance model that automatically tunes the retrieval quality-latency trade-off based on real-time generation state and underlying hardware characteristics

Evaluation Highlights

Up to 2.6x speedup in end-to-end generation latency compared to standard sequential RAG execution
Simultaneous improvement in generation quality rather than a pure latency-quality trade-off

Signal Assessment

6/10 The paper delivers a well-executed, practical algorithm-system co-design that meaningfully accelerates RAG inference without sacrificing quality, but it builds on established pipelining/systems techniques applied to a specific RAG bottleneck rather than introducing a fundamentally new capability or paradigm.

Methodology

Analyze the latency breakdown of periodic-retrieval RAG architectures to identify sequential retrieval-generation execution as the primary bottleneck
Design a pipeline parallelism scheme that overlaps retrieval operations for future context with ongoing generation using recent context
Introduce flexible retrieval intervals that adapt scheduling to align with pipeline stages and maximize concurrency
Build a performance model that predicts the latency and quality impact of different retrieval configurations given current generation state and hardware, and uses it to select settings automatically
Evaluate end-to-end generation latency and output quality against standard sequential RAG baselines

System Components

Pipeline Parallelism Engine

Overlaps retrieval computation with concurrent token generation so the model does not stall waiting for retrieval to complete.

Flexible Retrieval Interval Scheduler

Dynamically decides when to trigger retrieval based on pipeline progress and generation state, rather than fixed periodic intervals, to maximize overlap efficiency.

Performance Model

Estimates the latency and quality trade-offs of candidate retrieval configurations given hardware characteristics and generation state, automatically selecting settings that balance speed and retrieval accuracy.

Results

Metric	Baseline (Sequential RAG)	PipeRAG	Delta
End-to-end generation latency	1x (sequential retrieve-then-generate)	Up to 2.6x faster	Up to 2.6x speedup
Generation quality	Standard periodic-retrieval quality	Improved over baseline	Net quality gain, not just a trade-off

Key Takeaways

Retrieval latency in periodic-retrieval RAG systems can be substantially hidden by overlapping it with generation via pipeline parallelism, rather than treating retrieval as a blocking step.
Fixed retrieval intervals are suboptimal; making retrieval timing adaptive to pipeline and generation state improves both speed and quality.
Jointly optimizing retrieval algorithm decisions with system/hardware execution characteristics (algorithm-system co-design) is key to unlocking practical, low-latency RAG at scale.
The approach is especially relevant for RETRO-style or other architectures that retrieve repeatedly throughout generation, a growing pattern in long-context and agentic LLM pipelines.

Abstract

Retrieval-augmented generation (RAG) can enhance the generation quality of large language models (LLMs) by incorporating external token databases. However, retrievals from large databases can constitute a substantial portion of the overall generation time, particularly when retrievals are periodically performed to align the retrieved content with the latest states of generation. In this paper, we introduce PipeRAG, a novel algorithm-system co-design approach to reduce generation latency and enhance generation quality. PipeRAG integrates (1) pipeline parallelism to enable concurrent retrieval and generation processes, (2) flexible retrieval intervals to maximize the efficiency of pipeline parallelism, and (3) a performance model to automatically balance retrieval quality and latency based on the generation states and underlying hardware. Our evaluation shows that, by combining the three aforementioned methods, PipeRAG achieves up to 2.6× speedup in end-to-end generation latency while improving generation quality. These promising results showcase the effectiveness of co-designing algorithms with underlying systems, paving the way for the adoption of PipeRAG in future RAG systems.