MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
Problem Statement
Most multimodal LLMs process speech and text representations with the same parameters, even though the two modalities have very different statistical structure. This can cause interference and weaker modality-specific learning. Strong speech-text LLMs also often depend on proprietary data and closed training pipelines, which makes them hard to reproduce, and no fully open speech-text LLM had used an MoE backbone.
Key Novelty
- MAMoE architecture: deterministic, modality-aware routing that sends each token to a speech-specific or text-specific expert group based on its input type, rather than routing purely on learned token content.
- Shared experts alongside the modality-specific groups, so the model can learn modality-specialized patterns and still transfer information across modalities.
- A data-efficient pipeline that converts a pretrained MoE text LLM into a speech-text model: ASR and TTS post-training followed by speech-text instruction tuning, using only open-source datasets.
- A fully open release (model, training and inference code, training data), presented as the first open-source MoE-based speech-text LLM.
Evaluation Highlights
- Reported to consistently outperform models of comparable parameter count across ASR, TTS, audio language modeling, and spoken question answering.
- Ablations attribute significant gains in every tested domain to both the modality-specific routing and the shared experts.
Signal Assessment
Methodology
- Start from a pretrained MoE language model and add speech tokenization, so speech can be represented as discrete tokens in the LLM's input/output space.
- Restructure the MoE layers into MAMoE: partition experts into speech-specific and text-specific groups, route tokens by modality, and keep shared experts active for all tokens.
- Post-train on open-source ASR (speech→text) and TTS (text→speech) datasets to align the modalities and train the modality-specific experts.
- Fine-tune on a curated speech-text instruction dataset to support spoken QA and interactive speech-language tasks.
- Evaluate on ASR, TTS, audio language modeling, and spoken QA benchmarks against similar-size baselines, and run ablations on the routing and shared-expert components.
System Components
Uses each token's modality (speech or text) to restrict its routing to the matching expert group, which reduces cross-modal parameter interference.
Separate sets of FFN experts dedicated to speech tokens and to text tokens, capturing domain-specific representational patterns.
Experts that process tokens from all modalities, providing a common pathway for cross-modal information transfer and alignment.
An existing text MoE language model that supplies linguistic knowledge and is adapted rather than trained from scratch, which improves data efficiency.
Trains on paired speech-text data in both directions to establish speech understanding and generation abilities.
A curated, open-source instruction-tuning corpus that enables spoken question answering and instruction following across modalities.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| ASR (WER) | Comparable-size speech-text LLMs | MoST | Consistently better (lower WER); exact numbers not given in abstract |
| TTS quality | Comparable-size speech-text LLMs | MoST | Consistently better; exact numbers not given in abstract |
| Audio language modeling | Comparable-size speech-text LLMs | MoST | Outperforms baselines (qualitative) |
| Spoken question answering | Comparable-size speech-text LLMs | MoST | Outperforms baselines (qualitative) |
| Ablation: remove modality-specific routing | Standard MoE routing | MAMoE | Significant gains across all domains |
| Ablation: remove shared experts | Modality-only experts | MAMoE with shared experts | Significant gains across all domains |
Key Takeaways
- For multimodal MoE models, hard-routing tokens by modality into dedicated expert groups is a cheap structural prior that reduces interference between modalities with different statistics. It can be applied to other modality pairs (e.g., vision-text, audio-video).
- Keep shared experts when separating modalities. Pure modality isolation loses cross-modal transfer, and the ablations indicate that the shared pathway is needed for the gains.
- A pretrained text MoE LLM can be turned into a competitive speech-text model using only open data: ASR/TTS post-training followed by instruction tuning. The released code and data make this a practical, reproducible baseline for speech-LLM research.
Abstract
We present MoST (Mixture of Speech and Text), a novel multimodal large language model that seamlessly integrates speech and text processing through our proposed Modality-Aware Mixture of Experts (MAMoE) architecture. While current multimodal models typically process diverse modality representations with identical parameters, disregarding their inherent representational differences, we introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type. MAMoE simultaneously enhances modality-specific learning and cross-modal understanding through two complementary components: modality-specific expert groups that capture domain-specific patterns and shared experts that facilitate information transfer between modalities. Building on this architecture, we develop an efficient transformation pipeline that adapts the pretrained MoE language model through strategic post-training on ASR and TTS datasets, followed by fine-tuning with a carefully curated speech-text instruction dataset. A key feature of this pipeline is that it relies exclusively on fully accessible, open-source datasets to achieve strong performance and data efficiency. Comprehensive evaluations across ASR, TTS, audio language modeling, and spoken question answering benchmarks show that MoST consistently outperforms existing models of comparable parameter counts. Our ablation studies confirm that the modality-specific routing mechanism and shared experts design significantly contribute to performance gains across all tested domains. To our knowledge, MoST represents the first fully open-source speech-text LLM built on a Mixture of Experts architecture. \footnote{We release MoST model, training code, inference code, and training data at https://github.com/NUS-HPC-AI-Lab/MoST