Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards
Problem Statement
RL for agentic small LLMs is held back by two problems. Open-source agentic training data covers few task types and is easily solved, so it gives little learning signal. Real-world APIs lack diversity and are unstable, rate-limited and nondeterministic, which makes large-scale RL rollouts costly and unreliable. Without diverse, challenging and stable training environments, small models cannot close the agentic gap with large, expensive models.
Key Novelty
- Joint synthesis of novel tasks and full tool ecosystems by a strong teacher model, rather than relying on fixed real APIs or narrow existing datasets
- Deliberate rewriting of tasks into underspecified instructions, paired with an LLM user simulator holding user-private information, so agents must learn to ask clarifying questions
- Task-level rubric rewards built from required subgoals, required user-agent interactions and forbidden behaviors, giving denser and more process-aware signals than outcome-only rewards
- A mock tool system that returns stable, simulated tool responses, enabling reproducible large-scale RL rollouts
Evaluation Highlights
- Evaluated on 14 challenging datasets spanning math reasoning, search and tool use
- Small models trained on SYNTHAGENT data achieve substantial gains and outperform larger baseline models (the abstract reports no exact numbers)
Signal Assessment
Methodology
- Use a strong teacher LLM to generate diverse novel tasks together with matching tool ecosystems (tool schemas and behaviors)
- Rewrite the generated tasks into intentionally underspecified instructions, moving key details into user-private information
- Build the simulated environment: an LLM-based user simulator answers the agent's clarification queries using the private information, and a mock tool system returns stable, consistent tool outputs
- Construct task-level rubrics listing required subgoals, expected user-agent interactions and forbidden behaviors
- Run RL on small LLMs in the simulated environments, using rubric-based evaluation (likely LLM-judged) as the reward signal
- Evaluate the trained models on 14 out-of-domain benchmarks in math, search and tool use against larger baselines
System Components
Strong LLM that invents diverse tasks and the tool ecosystems needed to solve them, broadening task variety beyond existing open-source data
Rewrites complete task specifications into ambiguous instructions so the agent must actively ask the user for missing details
Plays the user role, holds the private information withheld from the instruction, and answers the agent's clarifying questions
Simulates tool and API responses consistently and stably, avoiding the cost, latency and instability of real APIs during RL rollouts
Scores trajectories against task-level rubrics covering required subgoals, necessary user interactions and forbidden behaviors
Optimizes small agentic LLMs on the synthetic tasks within the simulated environment using the rubric rewards
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Math reasoning benchmarks | Small model trained on existing open-source agentic data | Small model trained on SYNTHAGENT data | Substantial gains (qualitative) |
| Search / agentic retrieval benchmarks | Small model baselines and larger models | SYNTHAGENT-trained small model | Substantial gains; competitive with or better than larger models |
| Tool-use benchmarks | Larger baseline models | SYNTHAGENT-trained small model | Small model outperforms larger baselines (qualitative) |
| Overall (14 datasets) | Larger baseline models | SYNTHAGENT-trained small models | Consistent improvements across domains |
Key Takeaways
- Fully simulated environments (mock tools plus LLM user simulator) can replace real APIs for agentic RL, giving stable, cheap and scalable rollouts without sacrificing transfer to real benchmarks
- Deliberately underspecifying training instructions is a simple, effective way to teach agents to ask clarifying questions, a skill outcome-only datasets rarely train
- Rubric rewards that encode subgoals, required interactions and forbidden behaviors give richer, safer RL signals than final-answer correctness and are especially useful for open-ended tool-use tasks
- Diversity and difficulty of synthetic tasks matter: teacher-generated novel tool ecosystems can help small models outperform larger models, making agentic capability more affordable to deploy
Abstract
Small LLMs often struggle to match the agentic capabilities of large, costly models. While reinforcement learning can help, progress has been limited by two structural bottlenecks: existing open-source agentic training data are narrow in task variety and easily solved; real-world APIs lack diversity and are unstable for large-scale reinforcement learning rollout processes. We address these challenges with SYNTHAGENT, a framework that jointly synthesizes diverse tool-use training data and simulates complete environments. Specifically, a strong teacher model creates novel tasks and tool ecosystems, then rewrites them into intentionally underspecified instructions. This compels agents to actively query users for missing details. When handling synthetic tasks, an LLM-based user simulator provides user-private information, while a mock tool system delivers stable tool responses. For rewards, task-level rubrics are constructed based on required subgoals, user-agent interactions, and forbidden behaviors. Across 14 challenging datasets in math, search, and tool use, models trained on our synthetic data achieve substantial gains, with small models outperforming larger baselines.