A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces
Problem Statement
Current RAG systems either retrieve once and concatenate passages, or follow a predefined workflow such as fixed iterative retrieval. Both keep retrieval decisions out of the model's hands. This wastes the reasoning and long-horizon tool-use abilities of modern LLMs and stops RAG quality from improving with model upgrades. It also leads to inefficient context use, because retrieval cannot adapt to how hard a question is or how fine-grained the needed evidence is.
Key Novelty
- An agentic RAG framework (A-RAG) that hands the retrieval process to the model through tool interfaces rather than hard-coded pipelines or prompt-scripted workflows.
- A hierarchical, multi-granularity retrieval interface with three tools: lexical keyword search, dense semantic search, and chunk-level reading. The agent can locate evidence coarsely, then read it at fine granularity.
- A systematic study of how agentic RAG performance scales with model size and test-time compute, framing RAG as a capability that should improve with stronger models.
- An open-sourced code base and evaluation suite for reproducible agentic RAG research.
Evaluation Highlights
- A-RAG consistently outperforms existing single-shot and workflow-based RAG approaches across multiple open-domain QA benchmarks. The abstract does not give exact benchmarks or numbers; multi-hop QA datasets are likely.
- Gains are achieved with comparable or fewer retrieved tokens, suggesting the agent retrieves more selectively and efficiently.
- A scaling analysis shows performance improving with larger backbone models and more test-time compute (e.g., more tool calls or reasoning steps).
Signal Assessment
Methodology
- Index the corpus at chunk level and build two retrieval backends: a lexical keyword index (e.g., BM25-style) and a dense embedding index for semantic search.
- Expose three tools to the LLM agent: keyword search, semantic search, and chunk read. The two search tools return compact candidates or snippets; chunk read retrieves full content for chosen chunks.
- Run the LLM in an agentic tool-use loop where it decides which tool to call, with what query, and when to stop. Evidence accumulates across steps before a final answer is produced.
- Evaluate on multiple open-domain QA benchmarks against single-shot RAG and workflow-based iterative RAG baselines, measuring answer quality and retrieved-token budget.
- Vary backbone model size and test-time compute budget to characterize scaling behavior.
System Components
Lexical retrieval for exact-match queries such as entities, rare terms, and identifiers, where dense retrievers often fail.
Dense embedding-based retrieval for paraphrased or conceptual queries, returning semantically related chunks.
Fine-grained access that lets the agent read the full content of selected chunks. This supports locate-then-read behavior and avoids dumping all retrieved text into context.
The frontier LLM that plans retrieval, picks tools and queries, judges evidence sufficiency, and decides when to answer.
Released benchmark harness for comparing agentic and non-agentic RAG on accuracy and retrieved-token efficiency.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Open-domain QA accuracy (multiple benchmarks) | Single-shot RAG (retrieve-and-concatenate) | A-RAG | Consistently higher (exact numbers not in abstract) |
| Open-domain QA accuracy (multiple benchmarks) | Workflow-based iterative RAG | A-RAG | Consistently higher (exact numbers not in abstract) |
| Retrieved tokens per query | Existing RAG approaches | A-RAG | Comparable or lower |
| Scaling with model size | Fixed pipelines (limited benefit from stronger models) | A-RAG | Performance improves with larger models (qualitative) |
| Scaling with test-time compute | Fixed retrieval budget | A-RAG | Performance improves with more compute (qualitative) |
Key Takeaways
- With strong tool-using LLMs, replace rigid RAG pipelines with a small set of well-designed retrieval tools and let the model drive retrieval. Your RAG system then improves for free when you upgrade the model.
- Offer multiple retrieval granularities and modalities: lexical and dense search for locating evidence, plus a separate read tool for full content. This hybrid, hierarchical interface balances recall with context efficiency.
- Agentic retrieval need not cost more context. Selective search-then-read can match or beat baselines on accuracy while using similar or fewer retrieved tokens, though latency and number of LLM calls should still be monitored.
- Treat test-time compute (number of tool calls or reasoning steps) as a tunable knob for RAG quality, and benchmark across model sizes before choosing a deployment configuration.
Abstract
Frontier language models have demonstrated strong reasoning and long-horizon tool-use capabilities. However, existing RAG systems fail to leverage these capabilities. They still rely on two paradigms: (1) designing an algorithm that retrieves passages in a single shot and concatenates them into the model's input, or (2) predefining a workflow and prompting the model to execute it step-by-step. Neither paradigm allows the model to participate in retrieval decisions, preventing efficient scaling with model improvements. In this paper, we introduce A-RAG, an Agentic RAG framework that exposes hierarchical retrieval interfaces directly to the model. A-RAG provides three retrieval tools: keyword search, semantic search, and chunk read, enabling the agent to adaptively search and retrieve information across multiple granularities. Experiments on multiple open-domain QA benchmarks show that A-RAG consistently outperforms existing approaches with comparable or lower retrieved tokens, demonstrating that A-RAG effectively leverages model capabilities and dynamically adapts to different RAG tasks. We further systematically study how A-RAG scales with model size and test-time compute. We will release our code and evaluation suite to facilitate future research. Code and evaluation suite are available at https://github.com/Ayanami0730/arag.