← Back to Paper Radar

Learning to Reason in 13 Parameters

John X. Morris, Niloofar Mireshghallah, Mark Ibrahim, Saeed Mahloujifar
arXiv.org | 2026
Learning to reason with RL may need only a tiny amount of new information. TinyLoRA, a parameterization that pushes low-rank adapters below rank 1 down to as few as one trainable parameter, recovers most RL reasoning gains, reaching 91% on GSM8K with an 8B Qwen2.5 model using just 13 bf16 parameters (26 bytes).

Problem Statement

Parameter-efficient fine-tuning such as LoRA is widely used for reasoning-oriented RL, but even rank-1 LoRA scales with the model's hidden dimension, so it cannot go below thousands to millions of parameters. That floor leaves open a basic question: how much capacity or information does an RL update for reasoning actually need? The answer matters for understanding what RL changes in a model and for extreme-efficiency settings such as serving many per-user or per-task adapters.

Key Novelty

  • TinyLoRA: a low-rank adapter parameterization that breaks the rank-1 / model-dimension floor of standard LoRA and can scale down to a single trainable parameter.
  • Empirical evidence that RL-based reasoning gains can be captured with extremely few parameters: 13 parameters reach 91% GSM8K accuracy on an 8B Qwen2.5 model, and about 90% of gains are recovered with roughly 1000x fewer parameters on harder math benchmarks.
  • A clear RL vs. SFT contrast in parameter efficiency: SFT needs 100-1000x larger updates to match RL performance, suggesting RL updates are far more information-sparse.

Evaluation Highlights

  • GSM8K: 91% accuracy for the 8B Qwen2.5 model with only 13 trained parameters (26 bytes in bf16).
  • AIME, AMC and MATH500: about 90% of the full fine-tuning performance improvement is recovered while training about 1000x fewer parameters. SFT requires 100-1000x larger updates to reach comparable performance.

Signal Assessment

7/10 The method is an incremental extension of LoRA-style adapters, but showing that RL reasoning gains fit in tens of parameters is a striking result. It sharpens the growing view that RL mostly elicits latent capabilities rather than teaching new knowledge, and it has real implications for interpretability and adapter efficiency.

Methodology

  1. Start from a pretrained base model (an 8B-scale Qwen2.5) and freeze all original weights.
  2. Reparameterize the adapter updates with TinyLoRA so the number of trainable parameters is decoupled from the model dimension and can range from one parameter upward. The abstract does not give the mechanism; plausible ingredients include frozen low-rank or SVD-derived bases, projection of a tiny trainable vector into adapter space, and parameter sharing across layers.
  3. Train the tiny adapter with reinforcement learning on math reasoning tasks, likely a GRPO-style policy-gradient method with verifiable rewards.
  4. Sweep the trainable-parameter budget across orders of magnitude and compare against standard LoRA and full fine-tuning on GSM8K, MATH500, AMC and AIME.
  5. Repeat the parameter-budget sweep with SFT to measure how update size requirements differ between RL and supervised training.

System Components

TinyLoRA adapter

An ultra-compact adapter parameterization that extends low-rank adaptation below rank 1, so the trainable parameter count can be set to arbitrarily small values, down to one.

Frozen base model (Qwen2.5, 8B scale)

The pretrained LLM whose latent reasoning ability is unlocked by the tiny adapter update.

RL training loop

Reinforcement learning with reward on verifiable answers, used to optimize the tiny parameter set. It proves far more parameter-efficient than supervised fine-tuning.

SFT comparison baseline

Supervised fine-tuning under the same parameter budgets, used to show that imitation learning needs 100-1000x more parameters to match RL.

Math reasoning benchmark suite

GSM8K, MATH500, AMC and AIME, used to measure how much reasoning improvement survives extreme parameter reduction.

Results

Metric/Benchmark Baseline This Paper Delta
GSM8K accuracy (Qwen2.5, 8B) Untuned base model (value not stated in abstract) 91% with 13 trained params (26 bytes) Large gain from a 26-byte update
AIME / AMC / MATH500 improvement recovered Full fine-tuning / standard LoRA (100% of gains) About 90% of gains About 1000x fewer trainable parameters
Update size needed: RL vs. SFT SFT RL with TinyLoRA SFT needs 100-1000x larger updates for equal performance
Minimum adapter size Rank-1 LoRA (scales with model dimension) TinyLoRA (down to 1 parameter) Orders of magnitude smaller

Key Takeaways

  • RL for reasoning appears to change very little information in the model and mostly elicits capabilities already present. Practitioners can likely use far smaller adapters than standard LoRA ranks for RL fine-tuning.
  • Because tiny adapters (tens of bytes) can carry meaningful behavioral changes, per-user or per-task adapters become nearly free to store, transmit and swap. The same property raises new security and auditing concerns about how little it takes to alter model behavior.
  • When choosing between SFT and RL under tight parameter or communication budgets (e.g., federated or edge settings), prefer RL: SFT needs 100-1000x more capacity to reach comparable reasoning gains.
  • Strong results on GSM8K-style tasks with almost no trainable parameters suggest checking whether benchmark gains reflect new skills or format and behavior elicitation, especially for Qwen-family models.

Abstract

Recent research has shown that language models can learn to \textit{reason}, often via reinforcement learning. Some work even trains low-rank parameterizations for reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91\% accuracy on GSM8K with only 13 trained parameters in bf16 (26 total bytes). We find this trend holds in general: we are able to recover 90\% of performance improvements while training $1000x$ fewer parameters across a suite of more difficult learning-to-reason benchmarks such as AIME, AMC, and MATH500. Notably, we are only able to achieve such strong performance with RL: models trained using SFT require $100-1000x$ larger updates to reach the same performance.

Generated automatically from available metadata and abstract on 2026-11-09.