← Back to Paper Radar

Online-Optimized RAG for Tool Use and Function Calling

Yu Pan, Xiaochen Li, Hanzhao Wang
arXiv.org | 2025
Online-Optimized RAG fixes embedding misalignment in tool and function retrieval by updating retrieval embeddings online during deployment. It learns from minimal feedback such as task success, using lightweight gradient steps, and leaves the LLM untouched.

Problem Statement

Tool-use and function-calling systems often depend on embedding similarity between user queries and tool descriptions. Imperfect embedding models or noisy, poorly written descriptions cause systematic misretrieval, so the agent picks the wrong tool and the task fails. Fixing this usually means offline re-training, curated labels, or LLM fine-tuning, none of which adapts to live traffic or to changing tool inventories.

Key Novelty

  • A deployment-time online learning framework that keeps adapting retrieval embeddings from live interactions, using sparse or bandit-style feedback such as task success instead of labeled query-tool pairs.
  • A plug-and-play design that needs no LLM changes and works across single- and multi-hop tool use, dynamic (growing or changing) tool inventories, and top-K retrieval with re-ranking.
  • A problem-dependent theoretical analysis (likely regret-style bounds) showing how performance depends on the quality of the initial embeddings and related problem quantities.

Evaluation Highlights

  • Consistent gains in tool selection accuracy over static-embedding RAG baselines across diverse tool-use benchmarks.
  • Higher end-task success rates, plus improvements that extend to general document-retrieval scenarios, with negligible added per-query latency.

Signal Assessment

5/10 This is a solid, practical contribution: it applies online learning under partial feedback to an important and underexplored failure mode in agentic tool retrieval, and backs it with theory. The underlying ideas (online embedding adaptation and bandit-style updates) are well established, so the advance is in the clean formulation and deployability rather than a conceptual leap.

Methodology

  1. Initialize query and tool/document embeddings from a pre-trained embedding model and retrieve the top-K tools by similarity (optionally followed by re-ranking).
  2. Run the LLM agent with the retrieved tool(s) and observe minimal feedback from the interaction, such as task success or failure (or per-hop signals in multi-hop settings).
  3. Apply a lightweight online gradient update to the retrieval embeddings (e.g., the tool embeddings or a projection/adapter) from this feedback, treating retrieval as a sequential decision problem with partial feedback.
  4. Handle dynamic inventories by adding new tool embeddings on the fly, and continue the updates as part of normal serving with negligible latency.
  5. Analyze convergence and regret as a function of the initial embedding alignment quality and other problem-dependent quantities.

System Components

Embedding-based retriever

Matches embedded user queries to tool/function or document description embeddings to pick candidates for the LLM.

Feedback collector

Captures minimal, possibly sparse signals from live interactions, such as task success, correctness of the chosen tool, or per-hop outcomes.

Online gradient updater

Performs cheap incremental updates to retrieval embeddings after each interaction to correct misalignment, without touching the LLM.

K-retrieval and re-ranking support

Extends the update scheme to settings where multiple candidates are retrieved and then re-ranked before use.

Multi-hop and dynamic inventory handling

Assigns feedback across sequential tool calls and accommodates tools added or removed at deployment time.

Problem-dependent theoretical analysis

Bounds performance in terms of initial embedding quality and problem structure, explaining when and how fast online adaptation helps.

Results

Metric/Benchmark Baseline This Paper Delta
Tool selection accuracy (tool-use benchmarks) Static embedding RAG Online-Optimized RAG Consistent improvement (no numbers in abstract)
End-task success rate Static embedding RAG Online-Optimized RAG Consistent improvement (no numbers in abstract)
Document retrieval quality Static embedding retrieval Online-Optimized RAG Improved (qualitative)
Per-query latency overhead None Negligible ≈0 added cost
LLM modification required N/A None Plug-and-play

Key Takeaways

  • Retrieval misalignment between queries and tool descriptions is a common, fixable cause of agent failures. Adapting the retriever online can beat re-engineering prompts or fine-tuning the LLM.
  • Even binary task-success feedback is enough to steadily improve tool retrieval. Instrument your agent pipelines to log outcomes so the retriever can self-improve in production.
  • Initial embedding quality still matters: online updates correct misalignment over time, but a better starting embedding model or cleaner tool descriptions speed convergence. Combine the two rather than relying on either alone.
  • The approach is compatible with growing tool catalogs and top-K plus re-ranker pipelines, so it can likely be added to existing function-calling stacks with little architectural change.

Abstract

In many applications, retrieval-augmented generation (RAG) drives tool use and function calling by embedding the (user) queries and matching them to pre-specified tool/function descriptions. In this paper, we address an embedding misalignment issue that often arises in practical applications due to imperfect embedding models or noisy descriptions; such misalignment may lead to incorrect retrieval and task failure. We introduce Online-Optimized RAG, a deployment-time framework that continually adapts retrieval embeddings from live interactions using minimal feedback (e.g., task success). Online-Optimized RAG applies lightweight online gradient updates with negligible per-query latency and requires no changes to the underlying LLM. The method is plug-and-play: it supports both single- and multi-hop tool use, dynamic tool inventories, and $K$-retrieval with re-ranking. We provide a problem-dependent theoretical analysis that quantifies how the method's performance depends on the initialization quality of the embeddings and other related quantities. Across diverse tool-use and document-retrieval scenarios, our Online-Optimized RAG consistently improves tool selection accuracy and end-task success, thus providing a simple, practical path to robust, self-improving RAG systems.

Generated automatically from available metadata and abstract on 2026-11-09.