EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens

EM-KD is a knowledge distillation framework for efficient MLLMs that first resolves the token-count mismatch between a compressed student and a vanilla teacher via Hungarian-matching-based vision token alignment, then transfers cross-modal and semantic knowledge through two tailored losses (VLAD and VSD).

Problem Statement

Efficient MLLMs compress vision tokens to cut compute costs, but this discards fine-grained visual information and degrades comprehension. Prior knowledge distillation approaches designed to recover this lost accuracy implicitly assume comparable teacher/student token sequences, overlooking that efficient students and vanilla teachers naturally produce unbalanced numbers of vision tokens, which makes direct token-to-token distillation ill-posed and limits its benefit.

Key Novelty

Formulates vision token alignment across mismatched teacher/student token counts as an assignment problem, using Manhattan distance as a cost function and the Hungarian algorithm to spatially match unbalanced tokens before any distillation loss is applied
Introduces Vision-Language Affinity Distillation (VLAD), transferring cross-modal text-vision relational structure by minimizing the smooth L1 distance between teacher and student text-to-vision affinity matrices
Introduces Vision Semantic Distillation (VSD), which exploits the semantic richness of final-layer vision logits by matching teacher/student vocabulary-space probability distributions via reverse KL divergence

Evaluation Highlights

Outperforms prior Efficient MLLM baselines by a large margin in accuracy across diverse benchmarks while preserving efficiency gains
Outperforms previous knowledge distillation methods even when those baselines are equipped with the paper's own vision-token matching strategy, isolating the added value of VLAD and VSD specifically

Signal Assessment

6/10 The paper delivers a well-motivated, technically elegant solution (assignment-based token alignment plus two new distillation objectives) to a concrete bottleneck in distilling efficient MLLMs, but it composes established techniques (Hungarian matching, KL/L1 distillation) rather than introducing a fundamentally new paradigm.

Methodology

Extract vision logits/representations from both the vanilla teacher (more vision tokens) and the efficient student (fewer, compressed vision tokens)
Compute a Manhattan distance cost matrix between teacher and student vision tokens and solve the corresponding Hungarian matching problem to spatially align student tokens with their closest teacher counterparts
Apply Vision-Language Affinity Distillation (VLAD): build text-to-vision affinity matrices for teacher and student over the aligned tokens and minimize their smooth L1 distance
Apply Vision Semantic Distillation (VSD): project aligned vision logits into vocabulary-space probability distributions and minimize the reverse KL divergence between teacher and student distributions
Jointly optimize the efficient student MLLM using the combined VLAD and VSD distillation losses alongside the base training objective

System Components

Vision Token Aligner (Manhattan distance + Hungarian matching)

Resolves the unbalanced vision token counts between teacher and student by solving an optimal spatial assignment problem, producing the correspondence required for downstream distillation

Vision-Language Affinity Distillation (VLAD)

Transfers cross-modal grounding knowledge by aligning the text-to-vision affinity matrix of the student with that of the teacher via a smooth L1 loss

Vision Semantic Distillation (VSD)

Transfers fine-grained semantic knowledge from the final-layer vision logits by matching teacher and student vocabulary-space probability distributions using reverse KL divergence

Results

Metric/Benchmark	Prior Efficient MLLM	EM-KD	Delta
Accuracy across diverse MLLM benchmarks	Baseline efficient MLLM	Higher accuracy	Large margin gain (exact figures not given in abstract)
Comparison vs. prior KD methods (with matching strategy applied for fairness)	Prior distillation accuracy	EM-KD accuracy	Consistent improvement, isolating VLAD/VSD contribution
Inference efficiency	Reduced vision tokens vs. vanilla model	Efficiency of compressed student preserved	Accuracy gain with no added inference cost

Key Takeaways

When distilling between models with mismatched vision token counts (e.g., due to token pruning/merging), explicitly align tokens via optimal assignment before applying feature-level distillation losses—naive positional matching is invalid
Cross-modal affinity matrices (text-vision) and vocabulary-space semantic distributions are effective, complementary distillation signals for MLLMs beyond raw hidden-state matching
Token compression in efficient MLLMs need not sacrifice accuracy if paired with a distillation strategy designed to handle unbalanced token structures
The proposed token alignment strategy is a general, pluggable component that can also improve existing KD methods, suggesting applicability beyond the specific VLAD/VSD losses introduced here

Abstract

Efficient Multimodal Large Language Models (MLLMs) compress vision tokens to reduce resource consumption, but the loss of visual information can degrade comprehension capabilities. Although some priors introduce Knowledge Distillation to enhance student models, they overlook the fundamental differences in fine-grained vision comprehension caused by unbalanced vision tokens between the efficient student and vanilla teacher. In this paper, we propose EM-KD, a novel paradigm that enhances the Efficient MLLMs with Knowledge Distillation. To overcome the challenge of unbalanced vision tokens, we first calculate the Manhattan distance between the vision logits of teacher and student, and then align them in the spatial dimension with the Hungarian matching algorithm. After alignment, EM-KD introduces two distillation strategies: 1) Vision-Language Affinity Distillation (VLAD) and 2) Vision Semantic Distillation (VSD). Specifically, VLAD calculates the affinity matrix between text tokens and aligned vision tokens, and minimizes the smooth L1 distance of the student and the teacher affinity matrices. Considering the semantic richness of vision logits in the final layer, VSD employs the reverse KL divergence to measure the discrete probability distributions of the aligned vision logits over the vocabulary space. Comprehensive evaluation on diverse benchmarks demonstrates that EM-KD trained model outperforms prior Efficient MLLMs on both accuracy and efficiency with a large margin, validating its effectiveness. Compared with previous distillation methods, which are equipped with our proposed vision token matching strategy for fair comparison, EM-KD also achieves better performance.