Improved Large Language Diffusion Models
Problem Statement
The LLM field is almost entirely built on autoregressive, causally-attended transformers, leaving open whether diffusion-based (bidirectional, non-causal) architectures can scale to comparable capability and size. Prior diffusion LMs like LLaDA demonstrated feasibility but lagged behind autoregressive models in benchmark performance, and diffusion models lack established best practices for efficient decoding and fair multiple-choice evaluation. This paper closes that gap by scaling training and introducing inference/evaluation techniques tailored to the diffusion paradigm.
Key Novelty
- First large-scale (8B params, 12T tokens) masked diffusion LLM trained from scratch with fully bidirectional attention across both pretraining and SFT stages
- Large-scale instruction tuning (25B tokens, 12 epochs) that preserves the masked diffusion objective rather than switching to an autoregressive fine-tuning scheme
- Variable-length generation strategy that improves inference efficiency for diffusion-based decoding
- Confidence-based scoring method enabling fairer multiple-choice benchmark evaluation for diffusion LMs
Evaluation Highlights
- iLLaDA-Base outperforms LLaDA by 21.6 points on BBH and 14.9 points on ARC-Challenge
- iLLaDA-Instruct outperforms LLaDA by 14.5 points on MATH and 16.5 points on HumanEval, with broad gains across general, math, and code benchmarks
- iLLaDA remains competitive with the autoregressive Qwen2.5-7B model on several benchmarks despite non-autoregressive training
Signal Assessment
Methodology
- Pretrain an 8B-parameter transformer with fully bidirectional (non-causal) attention using a masked diffusion objective on 12T tokens from scratch
- Perform supervised fine-tuning while retaining the same masked diffusion objective on a 25B-token instruction corpus over 12 epochs
- Apply variable-length generation during inference to reduce decoding cost/latency
- Introduce confidence-based scoring to adapt multiple-choice evaluation to the diffusion generation setting
- Benchmark against LLaDA (prior diffusion LM) and Qwen2.5-7B (strong autoregressive baseline) across general, mathematical, and code tasks
System Components
Training objective where input tokens are randomly masked and the model learns to iteratively denoise/predict them, used consistently in both pretraining and SFT instead of next-token prediction.
8B-parameter transformer backbone using full (non-causal) attention, enabling non-sequential, parallel token generation rather than strict left-to-right decoding.
Inference-time technique that adapts generation length dynamically to improve efficiency of the diffusion decoding process.
Evaluation method that leverages the model's token-level confidence estimates to score multiple-choice answers, tailored to how diffusion models produce outputs.
25B-token instruction-tuning dataset used for 12 epochs of fine-tuning while preserving the diffusion training objective.
Results
| Benchmark | LLaDA (Baseline) | iLLaDA (This Paper) | Delta |
|---|---|---|---|
| BBH (Base model) | LLaDA-Base score | LLaDA-Base score + 21.6 | +21.6 points |
| ARC-Challenge (Base model) | LLaDA-Base score | LLaDA-Base score + 14.9 | +14.9 points |
| MATH (Instruct model) | LLaDA-Instruct score | LLaDA-Instruct score + 14.5 | +14.5 points |
| HumanEval (Instruct model) | LLaDA-Instruct score | LLaDA-Instruct score + 16.5 | +16.5 points |
| Multiple benchmarks vs Qwen2.5-7B (AR) | Qwen2.5-7B | Competitive/comparable | ~Parity on several tasks |
Key Takeaways
- Masked diffusion LLMs can now be scaled to 8B parameters and 12T tokens while approaching parity with strong autoregressive models, making non-autoregressive architectures a credible alternative for practitioners to evaluate.
- Retaining the same diffusion objective throughout both pretraining and SFT simplifies the training pipeline and works effectively at scale, avoiding the need to hybridize with autoregressive fine-tuning.
- Diffusion LMs need dedicated inference and evaluation tooling (e.g., variable-length generation, confidence-based scoring) to be efficient and fairly comparable to autoregressive baselines.
- Open-sourced weights and code lower the barrier for researchers building agentic or multi-modal systems that could exploit diffusion LMs' parallel/bidirectional generation for tasks like infilling, editing, or structured output generation.
Abstract
Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present \emph{iLLaDA}, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.