← Back to Paper Radar

Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards

Yuan-Jay Lu, Chengyu Wang, Lei Shen, Jun Huang, Tong Xu
2026
SYNTHAGENT trains small agentic LLMs with RL on teacher-synthesized tasks, tool ecosystems and fully simulated environments: an LLM user simulator, mock tool responses and task-level rubric rewards. This replaces narrow open-source datasets and unstable real APIs, and the small models trained this way outperform larger baselines across math, search and tool-use benchmarks.

Problem Statement

RL for agentic small LLMs is held back by two problems. Open-source agentic training data covers few task types and is easily solved, so it gives little learning signal. Real-world APIs lack diversity and are unstable, rate-limited and nondeterministic, which makes large-scale RL rollouts costly and unreliable. Without diverse, challenging and stable training environments, small models cannot close the agentic gap with large, expensive models.

Key Novelty

  • Joint synthesis of novel tasks and full tool ecosystems by a strong teacher model, rather than relying on fixed real APIs or narrow existing datasets
  • Deliberate rewriting of tasks into underspecified instructions, paired with an LLM user simulator holding user-private information, so agents must learn to ask clarifying questions
  • Task-level rubric rewards built from required subgoals, required user-agent interactions and forbidden behaviors, giving denser and more process-aware signals than outcome-only rewards
  • A mock tool system that returns stable, simulated tool responses, enabling reproducible large-scale RL rollouts

Evaluation Highlights

  • Evaluated on 14 challenging datasets spanning math reasoning, search and tool use
  • Small models trained on SYNTHAGENT data achieve substantial gains and outperform larger baseline models (the abstract reports no exact numbers)

Signal Assessment

6/10 This is a solid, practical integration of known ideas (teacher-generated synthetic data, simulated users and tools, rubric-based LLM rewards) into a coherent RL pipeline for small agents, and the underspecification-plus-user-simulation design is a notable touch. Similar simulated-environment efforts exist, however, and the abstract gives no quantitative evidence of an outsized leap.

Methodology

  1. Use a strong teacher LLM to generate diverse novel tasks together with matching tool ecosystems (tool schemas and behaviors)
  2. Rewrite the generated tasks into intentionally underspecified instructions, moving key details into user-private information
  3. Build the simulated environment: an LLM-based user simulator answers the agent's clarification queries using the private information, and a mock tool system returns stable, consistent tool outputs
  4. Construct task-level rubrics listing required subgoals, expected user-agent interactions and forbidden behaviors
  5. Run RL on small LLMs in the simulated environments, using rubric-based evaluation (likely LLM-judged) as the reward signal
  6. Evaluate the trained models on 14 out-of-domain benchmarks in math, search and tool use against larger baselines

System Components

Teacher Task & Tool Synthesizer

Strong LLM that invents diverse tasks and the tool ecosystems needed to solve them, broadening task variety beyond existing open-source data

Underspecification Rewriter

Rewrites complete task specifications into ambiguous instructions so the agent must actively ask the user for missing details

LLM User Simulator

Plays the user role, holds the private information withheld from the instruction, and answers the agent's clarifying questions

Mock Tool System

Simulates tool and API responses consistently and stably, avoiding the cost, latency and instability of real APIs during RL rollouts

Rubric-Based Reward Model

Scores trajectories against task-level rubrics covering required subgoals, necessary user interactions and forbidden behaviors

RL Training Loop

Optimizes small agentic LLMs on the synthetic tasks within the simulated environment using the rubric rewards

Results

Metric/Benchmark Baseline This Paper Delta
Math reasoning benchmarks Small model trained on existing open-source agentic data Small model trained on SYNTHAGENT data Substantial gains (qualitative)
Search / agentic retrieval benchmarks Small model baselines and larger models SYNTHAGENT-trained small model Substantial gains; competitive with or better than larger models
Tool-use benchmarks Larger baseline models SYNTHAGENT-trained small model Small model outperforms larger baselines (qualitative)
Overall (14 datasets) Larger baseline models SYNTHAGENT-trained small models Consistent improvements across domains

Key Takeaways

  • Fully simulated environments (mock tools plus LLM user simulator) can replace real APIs for agentic RL, giving stable, cheap and scalable rollouts without sacrificing transfer to real benchmarks
  • Deliberately underspecifying training instructions is a simple, effective way to teach agents to ask clarifying questions, a skill outcome-only datasets rarely train
  • Rubric rewards that encode subgoals, required interactions and forbidden behaviors give richer, safer RL signals than final-answer correctness and are especially useful for open-ended tool-use tasks
  • Diversity and difficulty of synthetic tasks matter: teacher-generated novel tool ecosystems can help small models outperform larger models, making agentic capability more affordable to deploy

Abstract

Small LLMs often struggle to match the agentic capabilities of large, costly models. While reinforcement learning can help, progress has been limited by two structural bottlenecks: existing open-source agentic training data are narrow in task variety and easily solved; real-world APIs lack diversity and are unstable for large-scale reinforcement learning rollout processes. We address these challenges with SYNTHAGENT, a framework that jointly synthesizes diverse tool-use training data and simulates complete environments. Specifically, a strong teacher model creates novel tasks and tool ecosystems, then rewrites them into intentionally underspecified instructions. This compels agents to actively query users for missing details. When handling synthetic tasks, an LLM-based user simulator provides user-private information, while a mock tool system delivers stable tool responses. For rewards, task-level rubrics are constructed based on required subgoals, user-agent interactions, and forbidden behaviors. Across 14 challenging datasets in math, search, and tool use, models trained on our synthetic data achieve substantial gains, with small models outperforming larger baselines.

Generated automatically from available metadata and abstract on 2026-11-09.