MiMo-Audio: Audio Language Models are Few-Shot Learners
Problem Statement
Current audio language models usually need task-specific fine-tuning or narrowly scoped instruction data for each audio capability. Humans, by contrast, pick up new audio tasks from a few examples or a simple instruction. The field lacks evidence that massive-scale audio pretraining produces the general-purpose, in-context generalization seen in text LLMs, and it lacks open models and evaluation suites that systematically measure such abilities.
Key Novelty
- Scales audio-text next-token pretraining to more than 100M hours and reports emergent few-shot learning across diverse audio tasks, an audio analogue of the GPT-3 finding.
- Shows generalization to tasks absent from training data, such as voice conversion, style transfer and speech editing, along with strong long-form speech continuation (talk shows, debates, livestreams, recitations).
- Builds a systematic evaluation suite for few-shot and in-context audio capabilities, released openly.
- Adds 'thinking' (reasoning) mechanisms to both audio understanding and audio generation during instruction tuning.
- Releases open checkpoints for a Base model and an Instruct model, the latter competitive with closed-source systems.
Evaluation Highlights
- MiMo-Audio-7B-Base: open-source SOTA on speech-intelligence and audio-understanding benchmarks, with few-shot generalization to unseen tasks such as voice conversion, style transfer and speech editing.
- MiMo-Audio-7B-Instruct: open-source SOTA on audio understanding (MMSU, MMAU, MMAR, MMAU-Pro).
- MiMo-Audio-7B-Instruct: open-source SOTA on spoken dialogue (Big Bench Audio, MultiChallenge Audio).
- MiMo-Audio-7B-Instruct: open-source SOTA on instruct-TTS, approaching or surpassing closed-source models.
Signal Assessment
Methodology
- Discretize audio with a high-fidelity audio tokenizer. Per the full paper/repo, this is a residual-vector-quantized tokenizer at roughly 25 Hz with multiple codebook layers, designed to preserve both semantic and acoustic information.
- Use a patch-based architecture. Per the full paper/repo, a patch encoder groups consecutive audio-token frames into lower-rate patches for a 7B LLM backbone (MiMo-7B), and a patch decoder autoregressively generates the full-rate multi-layer audio tokens.
- Pretrain with next-token prediction over interleaved audio and text on more than 100M hours of audio, then probe emergent few-shot and in-context abilities on a systematic evaluation suite.
- Post-train on a curated, diverse instruction-tuning corpus spanning understanding, dialogue and TTS, adding thinking/reasoning traces for both audio understanding and generation to produce MiMo-Audio-7B-Instruct.
System Components
Converts waveforms into discrete multi-codebook tokens (RVQ-style per the full paper) that keep semantic and acoustic detail, so the LLM can both understand and generate audio.
Aggregates several consecutive audio-token frames into a single patch representation. This lowers the sequence rate the LLM must model and improves efficiency on long audio.
Text-pretrained 7B transformer that models interleaved text and audio-patch sequences by next-token prediction, supplying in-context learning and reasoning.
Expands each LLM output patch autoregressively into full-rate, multi-layer audio tokens for high-fidelity speech generation.
More than 100M hours of audio, the scale at which few-shot abilities are reported to emerge.
Curated multi-task instruction data that includes reasoning traces for both audio understanding and audio generation, used to build the Instruct model.
Open benchmark suite measuring in-context and few-shot generalization across speech and audio tasks.
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Speech intelligence benchmarks (Base) | Prior open-source audio LMs | MiMo-Audio-7B-Base: open-source SOTA | Best among open-source (no figures in abstract) |
| Audio understanding benchmarks (Base) | Prior open-source audio LMs | MiMo-Audio-7B-Base: open-source SOTA | Best among open-source (no figures in abstract) |
| Unseen tasks: voice conversion, style transfer, speech editing (Base) | Task-specific fine-tuned models / no in-context ability | Achieved via few-shot in-context learning | New capability without fine-tuning |
| MMSU / MMAU / MMAR / MMAU-Pro (Instruct) | Open-source audio LMs; closed-source models | Open-source SOTA | Approaches or surpasses closed-source |
| Big Bench Audio / MultiChallenge Audio (Instruct) | Open-source spoken dialogue models; closed-source models | Open-source SOTA | Approaches or surpasses closed-source |
| Instruct-TTS evaluation (Instruct) | Open-source TTS / audio LMs; closed-source models | Open-source SOTA | Approaches or surpasses closed-source |
Key Takeaways
- Scale matters for audio as it did for text. On the order of 100M hours of next-token pretraining appears to unlock in-context learning, so practitioners can prototype new speech tasks (voice conversion, style transfer, editing) with few-shot prompts before investing in fine-tuning.
- Patch-based compression of audio tokens before a text LLM is a practical way to make long-form audio modeling tractable while keeping high-fidelity generation through a separate patch decoder. It is a useful design pattern for omni or speech agents.
- Adding reasoning ('thinking') to audio understanding and generation during instruction tuning improves complex audio QA, spoken dialogue and controllable TTS. This suggests chain-of-thought-style post-training carries over to multimodal speech systems.
- The open Base and Instruct checkpoints plus the evaluation suite give a strong 7B foundation for building speech agents, and a reproducible benchmark for measuring few-shot audio generalization.
Abstract
Existing audio language models typically rely on task-specific fine-tuning to accomplish particular audio tasks. In contrast, humans are able to generalize to new audio tasks with only a few examples or simple instructions. GPT-3 has shown that scaling next-token prediction pretraining enables strong generalization capabilities in text, and we believe this paradigm is equally applicable to the audio domain. By scaling MiMo-Audio's pretraining data to over one hundred million of hours, we observe the emergence of few-shot learning capabilities across a diverse set of audio tasks. We develop a systematic evaluation of these capabilities and find that MiMo-Audio-7B-Base achieves SOTA performance on both speech intelligence and audio understanding benchmarks among open-source models. Beyond standard metrics, MiMo-Audio-7B-Base generalizes to tasks absent from its training data, such as voice conversion, style transfer, and speech editing. MiMo-Audio-7B-Base also demonstrates powerful speech continuation capabilities, capable of generating highly realistic talk shows, recitations, livestreaming and debates. At the post-training stage, we curate a diverse instruction-tuning corpus and introduce thinking mechanisms into both audio understanding and generation. MiMo-Audio-7B-Instruct achieves open-source SOTA on audio understanding benchmarks (MMSU, MMAU, MMAR, MMAU-Pro), spoken dialogue benchmarks (Big Bench Audio, MultiChallenge Audio) and instruct-TTS evaluations, approaching or surpassing closed-source models. Model checkpoints and full evaluation suite are available at https://github.com/XiaomiMiMo/MiMo-Audio.