Learning to Reason in 13 Parameters
Problem Statement
Parameter-efficient fine-tuning such as LoRA is widely used for reasoning-oriented RL, but even rank-1 LoRA scales with the model's hidden dimension, so it cannot go below thousands to millions of parameters. That floor leaves open a basic question: how much capacity or information does an RL update for reasoning actually need? The answer matters for understanding what RL changes in a model and for extreme-efficiency settings such as serving many per-user or per-task adapters.
Key Novelty
- TinyLoRA: a low-rank adapter parameterization that breaks the rank-1 / model-dimension floor of standard LoRA and can scale down to a single trainable parameter.
- Empirical evidence that RL-based reasoning gains can be captured with extremely few parameters: 13 parameters reach 91% GSM8K accuracy on an 8B Qwen2.5 model, and about 90% of gains are recovered with roughly 1000x fewer parameters on harder math benchmarks.
- A clear RL vs. SFT contrast in parameter efficiency: SFT needs 100-1000x larger updates to match RL performance, suggesting RL updates are far more information-sparse.
Evaluation Highlights
- GSM8K: 91% accuracy for the 8B Qwen2.5 model with only 13 trained parameters (26 bytes in bf16).
- AIME, AMC and MATH500: about 90% of the full fine-tuning performance improvement is recovered while training about 1000x fewer parameters. SFT requires 100-1000x larger updates to reach comparable performance.
Signal Assessment
Methodology
- Start from a pretrained base model (an 8B-scale Qwen2.5) and freeze all original weights.
- Reparameterize the adapter updates with TinyLoRA so the number of trainable parameters is decoupled from the model dimension and can range from one parameter upward. The abstract does not give the mechanism; plausible ingredients include frozen low-rank or SVD-derived bases, projection of a tiny trainable vector into adapter space, and parameter sharing across layers.
- Train the tiny adapter with reinforcement learning on math reasoning tasks, likely a GRPO-style policy-gradient method with verifiable rewards.
- Sweep the trainable-parameter budget across orders of magnitude and compare against standard LoRA and full fine-tuning on GSM8K, MATH500, AMC and AIME.
- Repeat the parameter-budget sweep with SFT to measure how update size requirements differ between RL and supervised training.
System Components
An ultra-compact adapter parameterization that extends low-rank adaptation below rank 1, so the trainable parameter count can be set to arbitrarily small values, down to one.
The pretrained LLM whose latent reasoning ability is unlocked by the tiny adapter update.
Reinforcement learning with reward on verifiable answers, used to optimize the tiny parameter set. It proves far more parameter-efficient than supervised fine-tuning.
Supervised fine-tuning under the same parameter budgets, used to show that imitation learning needs 100-1000x more parameters to match RL.
GSM8K, MATH500, AMC and AIME, used to measure how much reasoning improvement survives extreme parameter reduction.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| GSM8K accuracy (Qwen2.5, 8B) | Untuned base model (value not stated in abstract) | 91% with 13 trained params (26 bytes) | Large gain from a 26-byte update |
| AIME / AMC / MATH500 improvement recovered | Full fine-tuning / standard LoRA (100% of gains) | About 90% of gains | About 1000x fewer trainable parameters |
| Update size needed: RL vs. SFT | SFT | RL with TinyLoRA | SFT needs 100-1000x larger updates for equal performance |
| Minimum adapter size | Rank-1 LoRA (scales with model dimension) | TinyLoRA (down to 1 parameter) | Orders of magnitude smaller |
Key Takeaways
- RL for reasoning appears to change very little information in the model and mostly elicits capabilities already present. Practitioners can likely use far smaller adapters than standard LoRA ranks for RL fine-tuning.
- Because tiny adapters (tens of bytes) can carry meaningful behavioral changes, per-user or per-task adapters become nearly free to store, transmit and swap. The same property raises new security and auditing concerns about how little it takes to alter model behavior.
- When choosing between SFT and RL under tight parameter or communication budgets (e.g., federated or edge settings), prefer RL: SFT needs 100-1000x more capacity to reach comparable reasoning gains.
- Strong results on GSM8K-style tasks with almost no trainable parameters suggest checking whether benchmark gains reflect new skills or format and behavior elicitation, especially for Qwen-family models.
Abstract
Recent research has shown that language models can learn to \textit{reason}, often via reinforcement learning. Some work even trains low-rank parameterizations for reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91\% accuracy on GSM8K with only 13 trained parameters in bf16 (26 total bytes). We find this trend holds in general: we are able to recover 90\% of performance improvements while training $1000x$ fewer parameters across a suite of more difficult learning-to-reason benchmarks such as AIME, AMC, and MATH500. Notably, we are only able to achieve such strong performance with RL: models trained using SFT require $100-1000x$ larger updates to reach the same performance.