OpenThoughts-Agent: Data Recipes for Agentic Models
Problem Statement
Agentic LLMs are rapidly expanding AI's real-world applications, but there is no public playbook for curating training data that produces agents which generalize across tasks rather than overfitting to one benchmark. Prior open efforts (SWE-Smith, SERA, Nemotron-Terminal) are narrowly scoped to single domains, so the field lacks reproducible, systematic guidance on data recipes for broadly capable agents.
Key Novelty
- First large-scale, fully open systematic study (100+ controlled ablations) isolating the effect of each stage of an agentic data curation pipeline, including task source selection and diversity
- A generalist 100K-example training set assembled from empirically validated pipeline choices, rather than single-benchmark-targeted data
- Demonstration of compute-controlled scaling advantages over alternative open datasets at every training set size, providing evidence the data quality gains are not just a fixed-size artifact
- Full open release of data, pipeline code, ablation logs, and trained models to enable reproducible agentic training research
Evaluation Highlights
- 44.8% average accuracy across seven agentic benchmarks after fine-tuning Qwen3-32B on the curated 100K dataset
- 3.9 percentage point improvement over the strongest existing open-data agentic model, Nemotron-Terminal-32B (40.9%)
- Outperforms alternative open agentic datasets at every training set size under compute-controlled comparisons
Signal Assessment
Methodology
- Decompose the agentic data curation pipeline into discrete stages (task sourcing, environment/trajectory generation, filtering, diversity selection)
- Run 100+ controlled ablation experiments varying task sources and diversity strategies to measure downstream impact on agentic performance
- Use ablation insights to assemble a final curated training set of 100K examples spanning diverse agentic task types
- Fine-tune Qwen3-32B on the curated dataset and evaluate on seven agentic benchmarks
- Perform compute-controlled scaling comparisons against alternative open datasets to validate data quality independent of dataset size
System Components
Aggregates and selects tasks from diverse agentic domains/environments rather than a single benchmark, to promote generalization
Runs 100+ controlled experiments to isolate the causal effect of each pipeline stage (source mix, diversity, filtering) on downstream agentic accuracy
Selects and balances the final 100K training examples based on empirically validated diversity and source-mix principles
Qwen3-32B trained via SFT on the curated dataset to act as a broadly capable agent
Seven agentic benchmarks used to measure generalization rather than single-task performance
Publicly available training sets, pipeline code, experimental ablation data, and model checkpoints hosted at openthoughts.ai
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Avg. accuracy across 7 agentic benchmarks | 40.9% (Nemotron-Terminal-32B) | 44.8% (OT-Agent, Qwen3-32B) | +3.9 pp |
| Compute-controlled scaling vs. open datasets | Lower accuracy at matched training set sizes | Higher accuracy at every training set size | Consistent advantage across scale |
| Systematic pipeline understanding | Largely undocumented/single-benchmark focused | 100+ controlled ablations across pipeline stages | New reproducible design insights |
Key Takeaways
- Diversity of task sources matters more than raw data volume for building agents that generalize across benchmarks, rather than overfitting to one domain
- Systematic, staged ablation of the data pipeline (sourcing, generation, filtering, diversity) is a practical methodology practitioners can replicate for their own agentic training efforts
- Open, well-curated data recipes can match or exceed narrowly-targeted proprietary or single-benchmark datasets, lowering the barrier to competitive open agentic models
- Compute-controlled scaling comparisons are important for validating that data quality gains persist as training set size grows, not just at one fixed scale
- Publicly released pipelines, ablation logs, and checkpoints (openthoughts.ai) provide a reusable foundation for future agentic model training research
Abstract
Agentic language models dramatically expand the applications of AI yet little is publicly known about how to curate training data for broadly capable agents. Existing open efforts such as SWE-Smith, SERA, and Nemotron-Terminal typically target a single benchmark, leaving open the question of how to train models that generalize across diverse agentic tasks. The OpenThoughts-Agent (OT-Agent) project addresses this gap with a fully open data curation pipeline for training agentic models. We conduct more than 100 controlled ablation experiments to systematically investigate each stage of the pipeline, yielding insights on the importance of task sources and diversity. We then assemble a training set of 100K examples from our pipeline and fine-tune Qwen3-32B on this dataset, which yields an average accuracy of 44.8% across seven agentic benchmarks and a 3.9 percentage point improvement over the strongest existing open data agentic model (Nemotron-Terminal-32B, 40.9%). Moreover, our training data exhibits strong scaling properties, outperforming alternative open datasets at every training set size in compute-controlled comparisons. We publicly release our training sets, data pipeline, experimental data, and models at openthoughts.ai to support future open research on agentic model training.