Recursive Language Models
Problem Statement
Even frontier LLMs have bounded context windows. Their quality also degrades on long inputs well before hitting the limit ("context rot"), especially on tasks that need dense reasoning across the whole input rather than simple retrieval. Common workarounds such as retrieval, summarization/compaction and fixed long-context scaffolds discard information or impose rigid decomposition strategies. A general, model-driven way to scale effective context at inference time is therefore valuable for agents, codebases, document corpora and other long-horizon settings.
Key Novelty
- Reframes long-context processing as inference-time scaling: the prompt lives outside the model's context as a manipulable environment variable rather than as input tokens.
- A general recursive inference paradigm in which the LLM writes code to examine and decompose the prompt and spawns recursive self-calls on snippets, letting the model choose its own decomposition strategy rather than following a hand-designed scaffold.
- The first natively recursive language model (RLM-Qwen3-8B), post-trained at small scale to operate in the recursive paradigm. This shows the behavior can be learned, not just prompted.
Evaluation Highlights
- RLMs successfully process inputs up to ~100x (two orders of magnitude) beyond the underlying model's context window. Even on prompts that fit in context, they substantially outperform vanilla frontier LLMs and common long-context scaffolds across four diverse long-context tasks at comparable cost.
- RLM-Qwen3-8B outperforms base Qwen3-8B by 28.3% on average and approaches vanilla GPT-5 quality on three long-context tasks.
Signal Assessment
Methodology
- Load the long prompt into an external programmable environment (e.g., a code/REPL-style workspace) as a variable, instead of placing it in the model's context window.
- Give the root LLM only the query and metadata about the environment. It then writes code to inspect, search, chunk and transform the prompt content.
- Allow the LLM to recursively call itself (or sub-LLM instances) on selected snippets or sub-tasks, aggregating intermediate results in the environment.
- Have the root model synthesize the final answer from the recursively computed results.
- Compare against vanilla frontier LLMs and common long-context scaffolds (e.g., retrieval- and summarization-based approaches) on four long-context tasks, measuring quality and cost.
- Post-train Qwen3-8B on recursive-inference behavior to produce RLM-Qwen3-8B, and evaluate it against its base model and GPT-5.
System Components
Stores the arbitrarily long input outside the context window as a programmatically accessible object the model can query and slice.
The top-level LLM that receives the task and orchestrates inspection, decomposition and aggregation by writing and executing code.
Sub-invocations of the LLM on prompt snippets or sub-problems, each operating within a manageable context size; results are returned to the environment.
Model-written code (e.g., filtering, chunking, searching, iterating) that determines how the prompt is broken down, replacing fixed hand-crafted scaffolds.
A small post-trained model that natively uses the recursive inference paradigm, demonstrating that recursive behavior can be learned.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Maximum processable input length | Model's native context window | Up to ~100x the context window | +2 orders of magnitude |
| Quality on four long-context tasks (frontier models) | Vanilla frontier LLMs and common long-context scaffolds | RLM-wrapped frontier LLMs | Dramatically higher (qualitative; abstract gives no exact numbers) |
| Inference cost | Vanilla LLMs and scaffolds | RLMs | Comparable |
| Average long-context performance (8B scale) | Qwen3-8B | RLM-Qwen3-8B | +28.3% average |
| Comparison to frontier model (3 tasks) | Vanilla GPT-5 | RLM-Qwen3-8B | Approaches GPT-5 quality |
Key Takeaways
- Treat very long inputs (codebases, document dumps, agent trajectories) as environment state the model queries programmatically, rather than stuffing them into context. This sidesteps context limits and mitigates quality degradation on long inputs.
- Letting the model choose its own decomposition via code and recursive self-calls can beat fixed retrieval or summarization scaffolds without raising cost. RLMs are worth trying as a drop-in inference wrapper before investing in longer-context models.
- Recursive inference is learnable. Even modest post-training lifts an 8B model substantially on long-context tasks, so training small models for recursive/agentic context management is a promising, cost-effective direction.
- In production, budget for variable latency and cost and add sandboxing. Recursive calls and code execution introduce non-deterministic call trees, so monitoring, recursion-depth limits and secure execution environments matter.
Abstract
We study allowing large language models (LLMs) to process arbitrarily long prompts through the lens of inference-time scaling. We propose Recursive Language Models (RLMs), a general inference paradigm that treats long prompts as part of an external environment and allows the LLM to programmatically examine, decompose, and recursively call itself over snippets of the prompt. We find that RLMs can successfully process inputs up to two orders of magnitude beyond model context windows and, even for shorter prompts, dramatically outperform the quality of vanilla frontier LLMs and common long-context scaffolds across four diverse long-context tasks while having comparable cost. At a small scale, we post-train the first natively recursive language model. Our model, RLM-Qwen3-8B, outperforms the underlying Qwen3-8B model by $28.3\%$ on average and even approaches the quality of vanilla GPT-5 on three long-context tasks. Code is available at https://github.com/alexzhang13/rlm.