← Back to Paper Radar

MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts

Yuxuan Lou, Kai Yang, Yang You
arXiv.org | 2026
MoST is a speech-text LLM built on a Modality-Aware Mixture of Experts (MAMoE). Speech and text tokens are routed to separate modality-specific expert groups, while shared experts carry information across modalities. The model is adapted from a pretrained MoE text LLM using only open-source ASR, TTS, and instruction data.

Problem Statement

Most multimodal LLMs process speech and text representations with the same parameters, even though the two modalities have very different statistical structure. This can cause interference and weaker modality-specific learning. Strong speech-text LLMs also often depend on proprietary data and closed training pipelines, which makes them hard to reproduce, and no fully open speech-text LLM had used an MoE backbone.

Key Novelty

  • MAMoE architecture: deterministic, modality-aware routing that sends each token to a speech-specific or text-specific expert group based on its input type, rather than routing purely on learned token content.
  • Shared experts alongside the modality-specific groups, so the model can learn modality-specialized patterns and still transfer information across modalities.
  • A data-efficient pipeline that converts a pretrained MoE text LLM into a speech-text model: ASR and TTS post-training followed by speech-text instruction tuning, using only open-source datasets.
  • A fully open release (model, training and inference code, training data), presented as the first open-source MoE-based speech-text LLM.

Evaluation Highlights

  • Reported to consistently outperform models of comparable parameter count across ASR, TTS, audio language modeling, and spoken question answering.
  • Ablations attribute significant gains in every tested domain to both the modality-specific routing and the shared experts.

Signal Assessment

5/10 This is a solid, practical contribution: modality-routed MoE with shared experts applied to speech-text, plus a fully open and reproducible pipeline. However, modality-aware experts and shared experts already exist in vision-language MoEs and DeepSeekMoE-style designs, so the architecture is more of a well-executed adaptation than a paradigm shift.

Methodology

  1. Start from a pretrained MoE language model and add speech tokenization, so speech can be represented as discrete tokens in the LLM's input/output space.
  2. Restructure the MoE layers into MAMoE: partition experts into speech-specific and text-specific groups, route tokens by modality, and keep shared experts active for all tokens.
  3. Post-train on open-source ASR (speech→text) and TTS (text→speech) datasets to align the modalities and train the modality-specific experts.
  4. Fine-tune on a curated speech-text instruction dataset to support spoken QA and interactive speech-language tasks.
  5. Evaluate on ASR, TTS, audio language modeling, and spoken QA benchmarks against similar-size baselines, and run ablations on the routing and shared-expert components.

System Components

Modality-Aware Router

Uses each token's modality (speech or text) to restrict its routing to the matching expert group, which reduces cross-modal parameter interference.

Modality-Specific Expert Groups

Separate sets of FFN experts dedicated to speech tokens and to text tokens, capturing domain-specific representational patterns.

Shared Experts

Experts that process tokens from all modalities, providing a common pathway for cross-modal information transfer and alignment.

Pretrained MoE LLM Backbone

An existing text MoE language model that supplies linguistic knowledge and is adapted rather than trained from scratch, which improves data efficiency.

ASR/TTS Post-Training Stage

Trains on paired speech-text data in both directions to establish speech understanding and generation abilities.

Speech-Text Instruction Dataset

A curated, open-source instruction-tuning corpus that enables spoken question answering and instruction following across modalities.

Results

Metric/Benchmark Baseline This Paper Delta
ASR (WER) Comparable-size speech-text LLMs MoST Consistently better (lower WER); exact numbers not given in abstract
TTS quality Comparable-size speech-text LLMs MoST Consistently better; exact numbers not given in abstract
Audio language modeling Comparable-size speech-text LLMs MoST Outperforms baselines (qualitative)
Spoken question answering Comparable-size speech-text LLMs MoST Outperforms baselines (qualitative)
Ablation: remove modality-specific routing Standard MoE routing MAMoE Significant gains across all domains
Ablation: remove shared experts Modality-only experts MAMoE with shared experts Significant gains across all domains

Key Takeaways

  • For multimodal MoE models, hard-routing tokens by modality into dedicated expert groups is a cheap structural prior that reduces interference between modalities with different statistics. It can be applied to other modality pairs (e.g., vision-text, audio-video).
  • Keep shared experts when separating modalities. Pure modality isolation loses cross-modal transfer, and the ablations indicate that the shared pathway is needed for the gains.
  • A pretrained text MoE LLM can be turned into a competitive speech-text model using only open data: ASR/TTS post-training followed by instruction tuning. The released code and data make this a practical, reproducible baseline for speech-LLM research.

Abstract

We present MoST (Mixture of Speech and Text), a novel multimodal large language model that seamlessly integrates speech and text processing through our proposed Modality-Aware Mixture of Experts (MAMoE) architecture. While current multimodal models typically process diverse modality representations with identical parameters, disregarding their inherent representational differences, we introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type. MAMoE simultaneously enhances modality-specific learning and cross-modal understanding through two complementary components: modality-specific expert groups that capture domain-specific patterns and shared experts that facilitate information transfer between modalities. Building on this architecture, we develop an efficient transformation pipeline that adapts the pretrained MoE language model through strategic post-training on ASR and TTS datasets, followed by fine-tuning with a carefully curated speech-text instruction dataset. A key feature of this pipeline is that it relies exclusively on fully accessible, open-source datasets to achieve strong performance and data efficiency. Comprehensive evaluations across ASR, TTS, audio language modeling, and spoken question answering benchmarks show that MoST consistently outperforms existing models of comparable parameter counts. Our ablation studies confirm that the modality-specific routing mechanism and shared experts design significantly contribute to performance gains across all tested domains. To our knowledge, MoST represents the first fully open-source speech-text LLM built on a Mixture of Experts architecture. \footnote{We release MoST model, training code, inference code, and training data at https://github.com/NUS-HPC-AI-Lab/MoST

Generated automatically from available metadata and abstract on 2026-11-09.