Training Language Agents to Learn from Experience

The paper trains a separate 'reflector' model via reinforcement learning to convert an actor agent's task trajectories into reusable system-prompt lessons that improve performance on future, unseen tasks—turning single-episode self-correction into genuine cross-task self-improvement.

Problem Statement

Existing reflection-based methods (e.g., Reflexion-style self-critique) only help an agent within the same task instance and don't produce knowledge that transfers to new tasks, limiting their practical value as agents encounter novel environments. This work matters because it tests whether the meta-skill of 'learning from experience' can itself be learned and generalized, rather than manually engineered, which is key for scalable, self-improving autonomous agents.

Key Novelty

Introduces In-context Training (ICT), a new task/framework that formalizes and evaluates cross-task self-improvement by having a reflector generate system prompts from actor trajectories for unseen future tasks
Proposes an RL-based training pipeline that teaches the reflector to produce effective lessons directly from experience, with no human-provided examples or supervision
Releases MetaGym, a general-purpose Python library for constructing meta-environments to support future research on self-improving agents
Demonstrates empirical evidence of held-out generalization, including some transfer to environments substantially different from the training benchmark

Evaluation Highlights

Trained reflectors outperform an untrained baseline reflector on most held-out task families within ALFWorld and MiniHack
Observed generalisation beyond the training benchmark to substantially different environments in some cases, suggesting transferable meta-strategies rather than narrow overfitting

Signal Assessment

6/10 The paper offers a solid, well-motivated extension of reflection-based agent methods by formalizing cross-task self-improvement and training it via RL, with promising but not yet dramatic generalization results, placing it above incremental but below a major paradigm shift.

Methodology

Define the ICT task: an actor model executes tasks and produces trajectories, which a reflector model observes to generate a system prompt intended to help on future unseen tasks
Construct training environments and task families using MetaGym to support meta-level train/held-out splits
Train the reflector with reinforcement learning, using downstream actor performance on subsequent tasks as the training signal, without human-authored reflection examples
Evaluate the trained reflector against an untrained baseline reflector across held-out task families in ALFWorld and MiniHack
Test cross-benchmark generalization by applying reflectors trained on one environment to substantially different environments

System Components

Actor model

Executes tasks in an interactive environment and produces trajectories that serve as the raw experience for reflection

Reflector model

Observes actor trajectories and generates a system prompt (reusable lesson) intended to improve the actor's performance on future, unseen tasks

In-context Training (ICT) task

A formal framework/benchmarking task for evaluating whether reflection-generated prompts transfer across tasks, not just within a single episode

RL-based training pipeline

Trains the reflector using feedback from downstream actor performance as reward, learning reflection generation directly from experience without human demonstrations

MetaGym

A generic Python library for constructing meta-environments (task families, held-out splits) to support research on self-improving language agents

Results

Benchmark	Untrained Reflector Baseline	Trained Reflector (This Paper)	Delta
ALFWorld (held-out task families)	Baseline system-prompt generation	Improved actor performance	Outperforms baseline on most held-out families
MiniHack (held-out task families)	Baseline system-prompt generation	Improved actor performance	Outperforms baseline on most held-out families
Cross-environment transfer	N/A / untested	Reflector trained on one benchmark applied to a different environment	Generalisation observed in some cases beyond training distribution

Key Takeaways

Self-improvement behaviors in language agents can be trained end-to-end with RL rather than relying solely on prompt engineering or fixed reflection heuristics
Formalizing reflection as a system-prompt-generation task (ICT) offers a concrete, measurable way to benchmark cross-task generalization of agent self-improvement methods
MetaGym lowers the barrier for practitioners to build custom meta-environments and held-out task splits for studying agent generalization
Observed transfer to substantially different environments suggests trained reflectors may capture generalizable meta-strategies, though robustness across broader domains still needs further validation
This approach could reduce reliance on manually crafted system prompts when deploying agents in new or evolving task domains

Abstract

Language agents can adapt from experience in interactive environments, but current reflection-based methods can only self-correct within a single task instance. Whether such experience can be distilled into reusable lessons that improve performance on future unseen tasks remains unclear. We address this problem by introducing the In-context Training (ICT) task, a framework for evaluating cross-task self-improvement in language agents. In ICT, a reflector model observes trajectories collected by an actor model and generates system prompts intended to improve the actor's performance on future unseen tasks. We then propose an RL-based training pipeline for learning such reflections directly from experience, without human-provided examples. Across ALFWorld and MiniHack, our trained reflectors outperform an untrained baseline on most held-out task families, showing that the ability to learn from experience can itself be learned. In some cases, we observe generalisation beyond the benchmark on which the reflector was trained, to substantially different environments. Finally, we introduce MetaGym, a generic Python library for constructing meta-environments, enabling future research on self-improving language agents.