Structured Reasoning for Large Language Models
Problem Statement
Long chain-of-thought reasoning improves accuracy but produces redundant steps, especially needless verification and revision after the model already has the correct answer. The authors attribute this to two causes: reasoning trajectories are unstructured, and training gives no targeted supervision for specific abilities such as self-verification and deciding when to terminate. The result is wasted inference compute and latency with no gain in quality.
Key Novelty
- A framework (SCR) that turns free-form reasoning traces into explicit, evaluable, trainable components, implemented mainly as a Generate-Verify-Revise paradigm.
- Dynamic Termination Supervision, which uses structured training data to teach the model when to stop reasoning instead of over-verifying correct answers.
- A progressive two-stage RL strategy that separates learning signals: Stage 1 optimizes initial generation and self-verification, and Stage 2 optimizes revision, so the objectives for different abilities do not interfere.
Evaluation Highlights
- Output token length drops by up to 50% compared with existing reasoning paradigms.
- Reasoning efficiency and self-verification improve substantially across three backbone models. The abstract does not give exact accuracy figures.
Signal Assessment
Methodology
- Construct structured training data that breaks reasoning trajectories into explicit Generate, Verify, and Revise segments with clear boundaries.
- Apply Dynamic Termination Supervision so the model learns to end reasoning once verification confirms a correct answer, and to revise only when needed.
- Run Stage 1 RL to optimize initial answer generation and self-verification accuracy.
- Run Stage 2 RL to optimize the revision ability, building on the Stage 1 policy so its learning signals do not interfere with earlier skills.
- Evaluate on three backbone LLMs against existing reasoning paradigms for accuracy, self-verification quality, and output token length.
System Components
Decouples reasoning traces into discrete, explicitly marked components that can each be evaluated and trained separately.
The main instantiation of SCR: produce an initial solution, self-verify it, and revise only if verification fails.
Supervision data formatted to match the component structure, enabling targeted learning of each reasoning ability.
Training signal that teaches the model when to stop reasoning, curbing redundant verification and revision after a correct answer.
Sequential reinforcement learning: Stage 1 targets generation and self-verification, and Stage 2 targets revision, avoiding gradient or reward interference between abilities.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Output token length | Existing reasoning paradigms (long CoT) | SCR | Up to 50% reduction |
| Reasoning efficiency | Unstructured long CoT | SCR | Substantially improved (qualitative) |
| Self-verification ability | Standard reasoning models | SCR | Substantially improved (qualitative) |
| Generalization across backbones | N/A | 3 backbone models | Consistent gains reported |
Key Takeaways
- Explicitly structuring reasoning traces (for example, with generate, verify, and revise segments) makes individual reasoning skills supervisable and can roughly halve inference token cost, which matters directly for serving budgets and latency.
- Teaching models when to stop is a high-leverage target. Much of the overthinking waste comes from verifying answers that are already correct, so termination-aware supervision is worth adding to reasoning RL pipelines.
- When training multiple reasoning abilities with RL, staging the objectives (generation and verification first, then revision) can avoid conflicting reward signals. This curriculum-style design transfers to agentic self-correction loops.
Abstract
Large language models (LLMs) achieve strong performance by generating long chains of thought, but longer traces always introduce redundant or ineffective reasoning steps. One typical behavior is that they often perform unnecessary verification and revisions even if they have reached the correct answers. This limitation stems from the unstructured nature of reasoning trajectories and the lack of targeted supervision for critical reasoning abilities. To address this, we propose Structured Reasoning (SCR), a framework that decouples reasoning trajectories into explicit, evaluable, and trainable components. We mainly implement SCR using a Generate-Verify-Revise paradigm. Specifically, we construct structured training data and apply Dynamic Termination Supervision to guide the model in deciding when to terminate reasoning. To avoid interference between learning signals for different reasoning abilities, we adopt a progressive two-stage reinforcement learning strategy: the first stage targets initial generation and self-verification, and the second stage focuses on revision. Extensive experiments on three backbone models show that SCR substantially improves reasoning efficiency and self-verification. Besides, compared with existing reasoning paradigms, it reduces output token length by up to 50%.