Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning
Problem Statement
Reasoning LLMs often produce excessively long chains of thought ("overthinking"). This raises inference cost and latency without improving accuracy, and sometimes hurts it. Existing efficient-reasoning methods tend to be complex, or they suppress useful self-verification along with redundant tokens. A simple, robust way to trade accuracy against length across diverse tasks is therefore needed.
Key Novelty
- A multi-stage efficient reasoning recipe: an SFT stage (rejection sampling for short correct traces, or reformatting of reasoning traces) followed by RL with an adaptive length penalty.
- A lightweight reward function that penalizes tokens generated after the first correct answer, while still rewarding self-verification when it is beneficial.
- A holistic evaluation of the accuracy versus response-length trade-off across seven diverse reasoning tasks, using the area under the Overthinking-Adjusted Accuracy curve (AUC_OAA) as a summary metric.
Evaluation Highlights
- Average response length falls by 28% for 8B models and 40% for 32B models, with accuracy drops of only 1.6 and 2.5 points respectively.
- AUC_OAA reaches 76.6: 5 points above the base model and 2.5 points above the second-best state-of-the-art efficient reasoning method.
Signal Assessment
Methodology
- Start from a reasoning-capable base LLM (8B and 32B scales) that tends to overthink.
- Stage 1 SFT: fine-tune on concise reasoning traces, built either by rejection sampling for short correct responses or by reformatting existing long traces to remove redundancy.
- Stage 2 RL: optimize with a reward that combines correctness and an adaptive length penalty targeting tokens generated after the first correct answer.
- Shape the reward so that self-verification is encouraged only when it improves correctness, which prevents collapse into brittle, unverified short answers.
- Evaluate on seven diverse reasoning tasks, measuring accuracy, response length and AUC_OAA against the base model and SOTA efficient-reasoning baselines.
System Components
Samples multiple responses from the model and keeps short, correct ones as fine-tuning targets, giving the model an initial bias toward concise reasoning.
An alternative SFT data source that restructures or condenses existing reasoning traces to remove redundant steps while keeping correctness.
A length-based term in the RL reward that adapts to the problem, penalizing excess computation rather than applying a fixed token budget.
A lightweight reward that penalizes tokens generated after the model first reaches the correct answer, directly targeting overthinking.
Keeps verification behavior when it actually helps the model reach or confirm a correct answer, so useful checking is not pruned away.
Area under the Overthinking-Adjusted Accuracy curve, a single scalar summarizing the accuracy versus length trade-off across budgets.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Avg. response length (8B) | Base model length (100%) | ~72% of base | -28% |
| Avg. accuracy (8B) | Base model accuracy | Base - 1.6 pts | -1.6 pts |
| Avg. response length (32B) | Base model length (100%) | ~60% of base | -40% |
| Avg. accuracy (32B) | Base model accuracy | Base - 2.5 pts | -2.5 pts |
| AUC_OAA vs. base model | 71.6 | 76.6 | +5.0 |
| AUC_OAA vs. second-best SOTA method | 74.1 | 76.6 | +2.5 |
Key Takeaways
- To cut inference costs of reasoning models, a simple pipeline (concise-trace SFT followed by RL with a length penalty) can deliver about 30–40% token savings with minimal accuracy loss. Elaborate efficient-reasoning machinery is not required.
- Penalizing tokens generated after the first correct answer is a cheap, targeted reward-shaping signal. It is easy to add to existing GRPO/PPO-style RL setups for reasoning models.
- Do not remove self-verification blindly. Rewarding verification only when it helps preserves robustness, and larger models appear to have more redundant reasoning to compress (40% vs. 28% length reduction).
- Evaluate efficiency methods on the joint accuracy–length trade-off (e.g., AUC_OAA) across diverse tasks, not on length or accuracy alone, to avoid misleading comparisons.
Abstract
The reasoning capabilities of large language models (LLMs) have improved substantially through increased test-time computation, typically in the form of intermediate tokens known as chain-of-thought (CoT). However, CoT often becomes unnecessarily long, increasing computation cost without actual accuracy gains or sometimes even degrading performance, a phenomenon known as ``overthinking''. We propose a multi-stage efficient reasoning method that combines supervised fine-tuning -- via rejection sampling or reasoning trace reformatting -- with reinforcement learning using an adaptive length penalty. We introduce a lightweight reward function that penalizes tokens generated after the first correct answer but encouraging self-verification only when beneficial. We conduct a holistic evaluation across seven diverse reasoning tasks, analyzing the accuracy-response length trade-off. Our approach reduces response length by an average of 28\% for 8B models and 40\% for 32B models, while incurring only minor performance drops of 1.6 and 2.5 points, respectively. Despite its conceptual simplicity, it achieves a superior trade-off compared to more complex state-of-the-art efficient reasoning methods, scoring 76.6, in terms of the area under the Overthinking-Adjusted Accuracy curve ($\text{AUC}_{\text{OAA}}$) -- 5 points above the base model and 2.5 points above the second-best approach.