In-Context Reinforcement Learning for Tool Use in Large Language Models
Problem Statement
Tool-augmented LLMs (e.g., code interpreters for math, search engines for factual QA) usually use a two-stage SFT→RL pipeline. The SFT stage exists mainly to teach tool-call formats and behaviors before RL can get useful rewards. That SFT stage needs costly annotated or synthesized tool-use trajectories, which limits scalability and makes it harder to extend to new tools or domains. ICRL instead uses in-context learning as the bootstrap, avoiding the data bottleneck.
Key Novelty
- An RL-only framework for tool use that replaces SFT cold-start with few-shot demonstrations injected into rollout prompts, using the model's in-context learning ability to produce valid tool-calling trajectories early in training.
- A curriculum that progressively reduces the number of in-context examples during training until it reaches zero-shot. This moves the model from imitating demonstrations in context to calling tools from internalized behavior, and narrows the train/inference prompt mismatch.
- Evidence that the approach is a data-efficient, scalable alternative to SFT+RL pipelines across both computational (Python interpreter) and knowledge-retrieval (search engine) tool settings.
Evaluation Highlights
- The abstract reports state-of-the-art performance across a range of reasoning and tool-use benchmarks, comparing ICRL against SFT-based cold-start pipelines. No specific numbers are given in the abstract.
- The method is claimed to reach this performance without labeled SFT data, positioning it as a data-efficient training recipe. The exact benchmarks (math reasoning vs. retrieval-augmented QA), model scales, and margins are not specified in the abstract.
Signal Assessment
Methodology
- Start from a base or instruction-tuned LLM with no tool-use SFT. Define a tool-calling protocol (e.g., tagged code or search calls whose outputs are executed and interleaved into the context).
- During RL rollouts, prepend N few-shot examples showing correct tool invocation and reasoning to each training prompt. This lets the model generate well-formed tool-use trajectories from the start, so reward signals are non-trivial.
- Score rollouts with a task reward (likely outcome-based correctness, possibly with format terms) and update the policy with a policy-gradient RL algorithm (a GRPO/PPO-style method is plausible but not stated in the abstract).
- Follow a curriculum schedule that gradually lowers the number of in-context examples as training progresses (e.g., N → N−1 → … → 0).
- Finish training in a zero-shot rollout setting so the final policy calls tools on its own at inference time, without demonstrations in the prompt.
- Evaluate zero-shot on reasoning and tool-use benchmarks against SFT→RL baselines.
System Components
Few-shot demonstrations of tool invocation inserted into rollout prompts. They act as a soft, data-light substitute for SFT, steering early exploration toward valid tool calls.
A schedule that decreases the number of in-context examples over training. It moves the policy from demonstration-conditioned behavior to independent zero-shot tool use and aligns training prompts with inference prompts.
External tools such as a Python interpreter for computation and a search engine for factual retrieval. Their outputs are fed back into the model's context during multi-turn rollouts.
A policy optimization stage that reinforces trajectories reaching correct answers, presumably via outcome or verifiable rewards. It is the only training signal, since no SFT stage is used.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Reasoning benchmarks with tool use (e.g., code-interpreter math) | SFT → RL cold-start pipelines | Reported SOTA (RL-only) | Improvement claimed; magnitude not stated in abstract |
| Tool-use / retrieval benchmarks (e.g., search-augmented QA) | SFT → RL cold-start pipelines | Reported SOTA (RL-only) | Improvement claimed; magnitude not stated in abstract |
| Labeled SFT data required | Substantial annotated/synthesized trajectories | None (few-shot prompts only) | Eliminates SFT data cost |
| Inference-time prompting | Zero-shot after SFT+RL | Zero-shot after curriculum annealing | Parity in deployment setting |
Key Takeaways
- If you are training a tool-using agent with RL and are blocked by a lack of SFT trajectories, try putting a handful of hand-written tool-use demonstrations in the rollout prompt. This can bootstrap non-zero reward without a cold-start dataset.
- Anneal the demonstrations rather than keeping them fixed. Removing in-context examples over training reduces dependence on the prompt scaffold and the mismatch between training and zero-shot deployment.
- Watch for subtle off-policy effects. Rollouts generated with demonstrations in context come from a slightly different conditional distribution than zero-shot inference, so check the schedule (step sizes, when to hit zero-shot) and confirm the final policy keeps tool-calling rates and formats without examples.
- The approach mainly benefits from base models with strong in-context learning, so gains may shrink for small models that cannot follow few-shot tool formats well. Validate on your target model scale before dropping SFT entirely.
Abstract
While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.