← Back to Paper Radar

MiMo-Audio: Audio Language Models are Few-Shot Learners

X. Zhang, Gang Wang, Jinlong Xue, Kai Fang, Liang Zhao, Rui Ma, Shu-Qin Ren, Shuo Liu, Tao Guo, Weiji Zhuang, Xin Zhang, Xi-Na Song, Yihan Yan, Yongzhe He, Cici, Bowen Shen, Chengxuan Zhu, Chong Ma, Chun Chen, Heyu Chen, Jiawei Li, Lei Li, Menghang Zhu, Peidian Li, Qi-ying Wang, Sirui Deng, Weimin Xiong, Wen Huang, Wenyu Yang, Yilin Jiang, Yixin Yang, Yu-Shi Tian, Yue Ma, Yue Yu, Zihan Zhang, Zihao Yue, Bangjun Xiao, Bin Xia, Bofei Gao, Bowen Ye, Can Cai, Chang Liu, Chenhong He, Chunan Li, Dawei Zhu, Duo Zhang, Fengyuan Shi, Guoan Wang, Hailin Zhang, Hanglong Lv, Hanyu Li, Hao Tian, Hengxu Qu, Hong-Mei Xu, Houbin Zhang, Huaqiu Liu, Jiangshan Duo, Jia Zuo, Jianyu Wei, Jiebao Xiao, Jinhao Dong, Jun Shi, Junhao Hu, Kainan Bao, Kang Zhou, Linghao Zhang, Meng Chen, Nuo Chen, Peng Zhang, Qian Chen, Qiantong Wang, Rang Li, Shao-yang Liu, Shengfan Wang, Shicheng Li, Shi-liang Yu, Shijie Cao, Shimao Chen, Shuhao Gu, Weikun Wang, Wen-Juan Ma, Xia Deng, Xing Yong, Xing Zhang, Xu Wang, Yi-Hao Song, Yihao Zhao, Yingbo Zhao, Yizhao Gao, Yu Cheng, Yuanfang Tu, Yudong Wang, Zhaojun Huang, Zheng-Yu Tang, Zhenrui Lin, Zhichao Song, Zhi-Yue Xu, Zhixian Zheng, Zi-Cheng Jiang
arXiv.org | 2025
Applying GPT-3-style scaling of next-token prediction pretraining to audio, using over 100 million hours of data, produces emergent few-shot in-context learning across diverse audio tasks. A 7B audio LM can therefore generalize to new speech and audio tasks without task-specific fine-tuning.

Problem Statement

Current audio language models usually need task-specific fine-tuning or narrowly scoped instruction data for each audio capability. Humans, by contrast, pick up new audio tasks from a few examples or a simple instruction. The field lacks evidence that massive-scale audio pretraining produces the general-purpose, in-context generalization seen in text LLMs, and it lacks open models and evaluation suites that systematically measure such abilities.

Key Novelty

  • Scales audio-text next-token pretraining to more than 100M hours and reports emergent few-shot learning across diverse audio tasks, an audio analogue of the GPT-3 finding.
  • Shows generalization to tasks absent from training data, such as voice conversion, style transfer and speech editing, along with strong long-form speech continuation (talk shows, debates, livestreams, recitations).
  • Builds a systematic evaluation suite for few-shot and in-context audio capabilities, released openly.
  • Adds 'thinking' (reasoning) mechanisms to both audio understanding and audio generation during instruction tuning.
  • Releases open checkpoints for a Base model and an Instruct model, the latter competitive with closed-source systems.

Evaluation Highlights

  • MiMo-Audio-7B-Base: open-source SOTA on speech-intelligence and audio-understanding benchmarks, with few-shot generalization to unseen tasks such as voice conversion, style transfer and speech editing.
  • MiMo-Audio-7B-Instruct: open-source SOTA on audio understanding (MMSU, MMAU, MMAR, MMAU-Pro).
  • MiMo-Audio-7B-Instruct: open-source SOTA on spoken dialogue (Big Bench Audio, MultiChallenge Audio).
  • MiMo-Audio-7B-Instruct: open-source SOTA on instruct-TTS, approaching or surpassing closed-source models.

Signal Assessment

7/10 This is a convincing, open demonstration that GPT-3-style scaling yields emergent in-context learning in audio, a significant step toward general-purpose audio foundation models. However, the recipe (scale plus next-token prediction plus instruction tuning) extends known text paradigms rather than introducing a fundamentally new method, and the abstract gives no quantitative margins.

Methodology

  1. Discretize audio with a high-fidelity audio tokenizer. Per the full paper/repo, this is a residual-vector-quantized tokenizer at roughly 25 Hz with multiple codebook layers, designed to preserve both semantic and acoustic information.
  2. Use a patch-based architecture. Per the full paper/repo, a patch encoder groups consecutive audio-token frames into lower-rate patches for a 7B LLM backbone (MiMo-7B), and a patch decoder autoregressively generates the full-rate multi-layer audio tokens.
  3. Pretrain with next-token prediction over interleaved audio and text on more than 100M hours of audio, then probe emergent few-shot and in-context abilities on a systematic evaluation suite.
  4. Post-train on a curated, diverse instruction-tuning corpus spanning understanding, dialogue and TTS, adding thinking/reasoning traces for both audio understanding and generation to produce MiMo-Audio-7B-Instruct.

System Components

MiMo-Audio Tokenizer

Converts waveforms into discrete multi-codebook tokens (RVQ-style per the full paper) that keep semantic and acoustic detail, so the LLM can both understand and generate audio.

Patch Encoder

Aggregates several consecutive audio-token frames into a single patch representation. This lowers the sequence rate the LLM must model and improves efficiency on long audio.

LLM Backbone (MiMo-7B)

Text-pretrained 7B transformer that models interleaved text and audio-patch sequences by next-token prediction, supplying in-context learning and reasoning.

Patch Decoder

Expands each LLM output patch autoregressively into full-rate, multi-layer audio tokens for high-fidelity speech generation.

Large-Scale Pretraining Corpus

More than 100M hours of audio, the scale at which few-shot abilities are reported to emerge.

Instruction-Tuning Corpus with Thinking

Curated multi-task instruction data that includes reasoning traces for both audio understanding and audio generation, used to build the Instruct model.

Few-Shot Audio Evaluation Suite

Open benchmark suite measuring in-context and few-shot generalization across speech and audio tasks.

Results

Metric/Benchmark Baseline This Paper Delta
Speech intelligence benchmarks (Base) Prior open-source audio LMs MiMo-Audio-7B-Base: open-source SOTA Best among open-source (no figures in abstract)
Audio understanding benchmarks (Base) Prior open-source audio LMs MiMo-Audio-7B-Base: open-source SOTA Best among open-source (no figures in abstract)
Unseen tasks: voice conversion, style transfer, speech editing (Base) Task-specific fine-tuned models / no in-context ability Achieved via few-shot in-context learning New capability without fine-tuning
MMSU / MMAU / MMAR / MMAU-Pro (Instruct) Open-source audio LMs; closed-source models Open-source SOTA Approaches or surpasses closed-source
Big Bench Audio / MultiChallenge Audio (Instruct) Open-source spoken dialogue models; closed-source models Open-source SOTA Approaches or surpasses closed-source
Instruct-TTS evaluation (Instruct) Open-source TTS / audio LMs; closed-source models Open-source SOTA Approaches or surpasses closed-source

Key Takeaways

  • Scale matters for audio as it did for text. On the order of 100M hours of next-token pretraining appears to unlock in-context learning, so practitioners can prototype new speech tasks (voice conversion, style transfer, editing) with few-shot prompts before investing in fine-tuning.
  • Patch-based compression of audio tokens before a text LLM is a practical way to make long-form audio modeling tractable while keeping high-fidelity generation through a separate patch decoder. It is a useful design pattern for omni or speech agents.
  • Adding reasoning ('thinking') to audio understanding and generation during instruction tuning improves complex audio QA, spoken dialogue and controllable TTS. This suggests chain-of-thought-style post-training carries over to multimodal speech systems.
  • The open Base and Instruct checkpoints plus the evaluation suite give a strong 7B foundation for building speech agents, and a reproducible benchmark for measuring few-shot audio generalization.

Abstract

Existing audio language models typically rely on task-specific fine-tuning to accomplish particular audio tasks. In contrast, humans are able to generalize to new audio tasks with only a few examples or simple instructions. GPT-3 has shown that scaling next-token prediction pretraining enables strong generalization capabilities in text, and we believe this paradigm is equally applicable to the audio domain. By scaling MiMo-Audio's pretraining data to over one hundred million of hours, we observe the emergence of few-shot learning capabilities across a diverse set of audio tasks. We develop a systematic evaluation of these capabilities and find that MiMo-Audio-7B-Base achieves SOTA performance on both speech intelligence and audio understanding benchmarks among open-source models. Beyond standard metrics, MiMo-Audio-7B-Base generalizes to tasks absent from its training data, such as voice conversion, style transfer, and speech editing. MiMo-Audio-7B-Base also demonstrates powerful speech continuation capabilities, capable of generating highly realistic talk shows, recitations, livestreaming and debates. At the post-training stage, we curate a diverse instruction-tuning corpus and introduce thinking mechanisms into both audio understanding and generation. MiMo-Audio-7B-Instruct achieves open-source SOTA on audio understanding benchmarks (MMSU, MMAU, MMAR, MMAU-Pro), spoken dialogue benchmarks (Big Bench Audio, MultiChallenge Audio) and instruct-TTS evaluations, approaching or surpassing closed-source models. Model checkpoints and full evaluation suite are available at https://github.com/XiaomiMiMo/MiMo-Audio.

Generated automatically from available metadata and abstract on 2026-11-09.