Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing
Problem Statement
Reliable tool/API invocation is essential for autonomous LLM agents, but current pipelines depend on costly human-annotated trajectories, generalize poorly to unseen tools, and inherit quality ceilings and biases when synthesized by a single model. This creates a scalability bottleneck that limits how well agentic LLMs can learn to use diverse real-world APIs. InfTool aims to remove human annotation entirely while avoiding the coverage gaps of single-model data generation.
Key Novelty
- A multi-agent role-play architecture (User Simulator, Tool-Calling Assistant, MCP Server) that synthesizes diverse, verified tool-use trajectories directly from raw API specs with no seed human data
- A self-evolving closed loop where the RL-improved model generates its own next-round training data, explicitly targeting its remaining capability gaps
- Integration of GRPO with gated (verification-conditioned) rewards to convert synthetic multi-agent trajectories into a reliable RL training signal without human labels
Evaluation Highlights
- BFCL accuracy on a 32B base model improves from 19.8% to 70.9% (+258% relative gain) using purely synthetic, self-generated data
- The resulting 32B model surpasses models roughly 10x larger in parameter count and performs comparably to Claude-Opus
Signal Assessment
Methodology
- Initialize the pipeline with only raw API/tool specifications, with no human-written trajectories
- Orchestrate three collaborative agents to co-generate trajectories: a User Simulator creates realistic queries/intents, a Tool-Calling Assistant selects and invokes tools, and an MCP Server executes/simulates tool responses and enables verification
- Filter and verify generated trajectories for correctness and diversity, covering single-turn calls through complex multi-step workflows
- Train the Tool-Calling Assistant on verified trajectories using GRPO with gated rewards tied to verification outcomes
- Redeploy the improved assistant to generate new, harder, higher-quality trajectories that specifically target its remaining capability gaps
- Repeat the generate-train-regenerate cycle autonomously, without any human intervention or annotation
System Components
An LLM agent that role-plays realistic users, generating diverse queries and intents that require tool invocation across varying complexity levels.
The agent under training that interprets user requests and selects/invokes the appropriate tools/APIs; this is the model iteratively improved by the closed loop.
A Model Context Protocol server that simulates or executes tool calls and returns responses, providing the environment feedback used to verify trajectory correctness.
Group Relative Policy Optimization variant where rewards are gated by verification signals, ensuring only correct/verified trajectories reinforce desired tool-use behavior.
The overall orchestration mechanism that cycles between data synthesis, RL training, and regeneration of higher-quality data targeting capability gaps, enabling autonomous continual improvement.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| BFCL accuracy (32B model) | 19.8% (base model) | 70.9% (InfTool-trained) | +258% relative (+51.1 pts absolute) |
| Model scale efficiency | Models ~10x larger | 32B InfTool model | Surpasses larger models |
| Comparison to proprietary SOTA | Claude-Opus | 32B InfTool model | Rivals/comparable accuracy |
Key Takeaways
- High-quality tool-use training data can be generated entirely autonomously via multi-agent role-play, eliminating the need for expensive human annotation of trajectories
- Closing the loop between synthetic data generation and RL training (GRPO) enables compounding self-improvement, where each iteration targets the model's current weaknesses
- A well-designed synthetic data and training pipeline can let a mid-sized (32B) model match or exceed the tool-calling performance of much larger or proprietary models like Claude-Opus
- Teams building agentic systems should consider MCP-based simulated tool environments as a scalable substitute for real API access when generating training/verification data
Abstract
Enabling Large Language Models (LLMs) to reliably invoke external tools remains a critical bottleneck for autonomous agents. Existing approaches suffer from three fundamental challenges: expensive human annotation for high-quality trajectories, poor generalization to unseen tools, and quality ceilings inherent in single-model synthesis that perpetuate biases and coverage gaps. We introduce InfTool, a fully autonomous framework that breaks these barriers through self-evolving multi-agent synthesis. Given only raw API specifications, InfTool orchestrates three collaborative agents (User Simulator, Tool-Calling Assistant, and MCP Server) to generate diverse, verified trajectories spanning single-turn calls to complex multi-step workflows. The framework establishes a closed loop: synthesized data trains the model via Group Relative Policy Optimization (GRPO) with gated rewards, the improved model generates higher-quality data targeting capability gaps, and this cycle iterates without human intervention. Experiments on the Berkeley Function-Calling Leaderboard (BFCL) demonstrate that InfTool transforms a base 32B model from 19.8% to 70.9% accuracy (+258%), surpassing models 10x larger and rivaling Claude-Opus, and entirely from synthetic data without human annotation.