← Back to Paper Radar

Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling

Falcon Llm Team, Iheb Chaabane, Puneesh Khanna, Suhail Mohmad, S. Frikha, Shi-Yu Huang, Abdalgader Abubaker, Réda Alami, Mikhail Lubinets, M. Seddik, Hakim Hacid
arXiv.org | 2026
Falcon-H1R is a 7B hybrid-architecture reasoning model. It shows that careful data curation, efficient SFT, and scaled RL let a small model match or beat reasoning models 2–7× larger. Its hybrid-parallel design and token efficiency also make it a strong backbone for compute-efficient test-time scaling, for example with DeepConf.

Problem Statement

Frontier reasoning gains have mostly come from scaling parameters and generating long chains of thought. Both make inference expensive, especially for parallel test-time scaling methods that sample many reasoning traces. Small models usually fall well short on reasoning-heavy benchmarks, and standard Transformers handle long CoT generation inefficiently because attention cost and KV-cache memory grow with sequence length. The paper targets all three efficiency axes together: accuracy, token usage, and inference speed.

Key Novelty

  • A 7B reasoning-optimized model that consistently matches or outperforms SOTA reasoning models 2–7× larger across reasoning-intensive benchmarks.
  • A focus on '3D reasoning efficiency', jointly optimizing accuracy, token efficiency, and inference throughput. Faster inference comes from a hybrid-parallel architecture, inherited from the Falcon-H1 family, which combines attention with state-space (Mamba-style) components.
  • A training recipe built on careful data curation, efficient SFT, and RL scaling, which delivers large reasoning gains without increasing model size.
  • Integration with DeepConf, a confidence-based method for filtering or weighting reasoning traces, achieving what the authors report as state-of-the-art test-time scaling efficiency in both accuracy and compute cost.

Evaluation Highlights

  • Matches or outperforms reasoning models 2–7× larger (roughly 14B–50B parameters) across a variety of reasoning-intensive benchmarks, likely math, code, and science-style tasks.
  • Under parallel test-time scaling with DeepConf, achieves improved accuracy at substantially lower computational cost. The gains come from higher throughput, fewer generated tokens, and pruning of low-confidence traces.

Signal Assessment

5/10 This is a solid, practically valuable contribution: it shows that hybrid SSM-attention architectures plus well-tuned SFT/RL can make small models competitive reasoners and efficient test-time-scaling backbones. However, the main ingredients (hybrid architecture, SFT+RL reasoning recipes, DeepConf) already exist, so the advance is mostly integration and engineering rather than a new paradigm.

Methodology

  1. Start from a hybrid-parallel base model (the Falcon-H1 7B backbone), which runs attention and state-space components in parallel for faster long-sequence inference.
  2. Curate high-quality reasoning data, such as long chain-of-thought traces in math, code, and science, with careful filtering and mixture design.
  3. Run efficient supervised fine-tuning (SFT) on the curated reasoning traces to instill long-form reasoning behavior.
  4. Apply RL scaling, likely with verifiable rewards, to further improve accuracy and token efficiency of reasoning chains.
  5. Evaluate against larger SOTA reasoning models on reasoning-intensive benchmarks, measuring accuracy, token usage, and inference speed.
  6. Combine the model with DeepConf for parallel test-time scaling, using model confidence signals to filter or weight sampled traces and trading off accuracy against compute.

System Components

Hybrid-parallel architecture

Combines attention with state-space (Mamba-style) layers in parallel. This reduces the memory and compute cost of long sequences and improves throughput for long CoT generation and batched parallel sampling.

Curated reasoning data pipeline

Selects and filters high-quality reasoning traces and problems. The paper identifies this as a key driver of performance gains at fixed model size.

Efficient SFT stage

Supervised fine-tuning on long reasoning traces to bootstrap chain-of-thought capability in the 7B model.

RL scaling stage

Reinforcement learning applied after SFT to push reasoning accuracy further while encouraging token-efficient reasoning.

DeepConf test-time scaling

A confidence-aware parallel sampling method that filters or early-stops low-confidence reasoning traces before aggregation. It improves accuracy per unit of compute compared with naive majority voting.

Results

Metric/Benchmark Baseline This Paper Delta
Reasoning benchmark accuracy (aggregate) SOTA reasoning models 2–7× larger (~14B–50B) Falcon-H1R-7B matches or exceeds Parity or better at 2–7× fewer parameters
Parameter count ~14B–50B 7B 2–7× smaller
Inference speed (long CoT) Standard Transformer reasoning models Faster via hybrid-parallel architecture Qualitatively higher throughput (no exact figure in abstract)
Token efficiency Comparable reasoning models Fewer tokens for similar or better accuracy Qualitative improvement
Test-time scaling (DeepConf) Other models under parallel sampling Reported SOTA accuracy–compute tradeoff Higher accuracy at substantially lower compute

Key Takeaways

  • For reasoning workloads that rely on many parallel samples (self-consistency, best-of-N, DeepConf), a small, fast, token-efficient model can beat a larger model on the accuracy-per-FLOP frontier. Measure total test-time compute, not just single-sample accuracy.
  • Hybrid SSM-attention architectures are a practical option for long chain-of-thought generation. They cut KV-cache and attention costs, which compound when generating many long traces.
  • Data curation plus a well-tuned SFT-then-RL pipeline can close much of the gap to larger reasoning models. Investing in reasoning data quality and RL scaling may pay off more than increasing parameter count for SLM deployments.
  • Confidence-based trace filtering such as DeepConf pairs well with efficient backbones. Consider adding it to inference stacks to lower cost without sacrificing accuracy.

Abstract

This work introduces Falcon-H1R, a 7B-parameter reasoning-optimized model that establishes the feasibility of achieving competitive reasoning performance with small language models (SLMs). Falcon-H1R stands out for its parameter efficiency, consistently matching or outperforming SOTA reasoning models that are $2\times$ to $7\times$ larger across a variety of reasoning-intensive benchmarks. These results underscore the importance of careful data curation and targeted training strategies (via both efficient SFT and RL scaling) in delivering significant performance gains without increasing model size. Furthermore, Falcon-H1R advances the 3D limits of reasoning efficiency by combining faster inference (through its hybrid-parallel architecture design), token efficiency, and higher accuracy. This unique blend makes Falcon-H1R-7B a practical backbone for scaling advanced reasoning systems, particularly in scenarios requiring extensive chain-of-thoughts generation and parallel test-time scaling. Leveraging the recently introduced DeepConf approach, Falcon-H1R achieves state-of-the-art test-time scaling efficiency, offering substantial improvements in both accuracy and computational cost. As a result, Falcon-H1R demonstrates that compact models, through targeted model training and architectural choices, can deliver robust and scalable reasoning performance.

Generated automatically from available metadata and abstract on 2026-11-09.