Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Models
Problem Statement
Aligning LLM and LVLM outputs with user intent is usually handled through costly training-stage interventions (e.g., RLHF, instruction tuning) that require significant compute and produce fixed, hard-to-adapt behavior post-deployment. Decoding methods offer a cheaper, model-agnostic, inference-time alternative, but prior work on these techniques is scattered across separate token-level, sequence-level, and multimodal literatures with no unifying framework. This survey addresses that gap by systematically organizing existing decoding strategies and clarifying their tradeoffs for both text-only and vision-language settings.
Key Novelty
- Introduces a three-paradigm taxonomy—token-level selection, sequence-level generation, and parallel/accelerated decoding—that unifies previously disconnected decoding techniques under one conceptual framework
- First survey to jointly treat decoding methods for LLMs and LVLMs, explicitly surfacing how multimodal grounding introduces distinct decoding challenges (e.g., hallucination mitigation) beyond text-only generation
- Delivers a practical, community-facing resource (curated GitHub repository) alongside a distilled set of open challenges and concrete future research directions
Evaluation Highlights
- Synthesizes and categorizes a large body of recent decoding literature into the three-paradigm taxonomy rather than reporting new empirical benchmark results
- Qualitatively compares decoding approaches against training-time alignment methods along efficiency, controllability, and scalability dimensions
Signal Assessment
Methodology
- Collected and reviewed recent literature on decoding methods spanning LLMs and LVLMs
- Categorized methods into three emerging paradigms based on the granularity and mechanism of control: token-level, sequence-level, and parallel generation
- Compared tradeoffs in efficiency, output quality, and controllability across categories, with dedicated attention to multimodal (LVLM) considerations
- Identified open challenges (e.g., standardized evaluation, efficiency-quality tradeoffs) and proposed future research directions
- Compiled and released a maintained public repository of papers and resources for ongoing community reference
System Components
Decoding strategies that steer generation by modifying next-token probabilities or logits at each step (e.g., contrastive decoding, guided/steered decoding, constrained sampling) for fine-grained, local control
Methods that search, sample, or evaluate entire candidate sequences or reasoning paths (e.g., beam search variants, self-consistency, tree/graph-structured search) to improve overall output quality and reasoning
Techniques that generate multiple tokens simultaneously (e.g., speculative decoding, non-autoregressive sampling) to reduce inference latency while preserving generation quality
Extensions of decoding strategies to vision-language models that address cross-modal grounding and visual hallucination issues not present in text-only decoding
Discussion of unresolved issues such as lack of standardized benchmarks, efficiency-quality tradeoffs, and generalization of decoding methods across tasks and modalities
Results
| Dimension | Training-time Alignment | Decoding-time Methods | Advantage |
|---|---|---|---|
| Cost | Requires retraining/fine-tuning (e.g., RLHF, SFT) | Inference-only, no parameter updates | Avoids expensive retraining cycles |
| Flexibility | Fixed behavior after training | Adjustable per task or query at inference time | Enables dynamic, task-specific control |
| Latency | Standard autoregressive decoding speed | Can be accelerated via parallel decoding paradigm | Reduces inference latency where applicable |
| Scope | Often modality- or task-specific tuning | Applicable across both LLMs and LVLMs | Broader, more generalizable applicability |
Key Takeaways
- Treat decoding-time methods (contrastive decoding, guided sampling, speculative decoding) as a lower-cost alternative or complement to RLHF/fine-tuning for controlling model behavior
- Match the decoding paradigm to your goal: token-level methods for fine-grained control, sequence-level/search methods for improving reasoning and quality, parallel decoding for cutting latency
- For LVLM deployments, adopt multimodal-aware decoding strategies to reduce hallucination and improve visual grounding rather than directly reusing text-only decoding techniques
- Be cautious of cross-paper comparisons given the field's lack of standardized decoding benchmarks; validate chosen methods on your own task-specific evaluation
- Use the accompanying GitHub repository as a living reference to track new decoding techniques in this fast-moving subfield
Abstract
Large language models (LLMs) and large vision-language models (LVLMs) have demonstrated impressive generative capabilities, yet ensuring their outputs align with user intent is still challenging. While most existing approaches address this issue at the training stage, inference-time approaches like decoding methods offer a more efficient and scalable solution. Decoding methods control model generation by guiding token-level selection, performing sequencelevel generation, or generating tokens in parallel to accelerate the process. In this survey, we identify three emerging paradigms from recent works on decoding methods for LLMs and LVLMs, provide a systematic review of these methods, highlight ongoing challenges, and discuss potential future research directions. Our goal is to underscore the efficiency and effectiveness of decoding methods and offer a practical view of their applications. Paper lists and more resources on decoding methods for LLMs and LVLMs can be found at https://github.com/wang2226/Awesome-LLM-Decoding.