AfriSpeech-MultiBench: A Verticalized Multidomain Multicountry Benchmark Suite for African Accented English ASR

The paper introduces AfriSpeech-MultiBench, the first verticalized, multi-domain, multi-country benchmark suite for evaluating ASR and multimodal speech recognition systems on African-accented English, spanning 100+ accents, 10+ countries, and seven application domains.

Problem Statement

Despite growing global adoption of voice AI (e.g., NotebookLM, speech-to-speech APIs), no application-specific evaluation framework exists for Africa's linguistically diverse English accents, leaving practitioners without guidance for deploying voice tech in African contexts. Generic ASR benchmarks fail to capture domain-specific challenges such as medical/legal terminology, named entities, and hallucination risks that are critical for real-world, high-stakes deployment in underserved communities.

Key Novelty

First domain-specific ('verticalized') ASR benchmark tailored to African-accented English, covering seven application verticals: Finance, Legal, Medical, General dialogue, Call Center, Named Entities, and Hallucination Robustness
Broad coverage combining 100+ accents across 10+ African countries with a diverse model pool spanning open-source, proprietary, unimodal ASR, and multimodal LLM-based speech recognition systems
Introduces dedicated evaluation axes—Named Entity accuracy and Hallucination Robustness—that are underexplored in standard ASR benchmarks but critical for enterprise/production use cases
Comparative analysis across spontaneous vs. non-spontaneous speech conditions, revealing model-type-specific degradation patterns not previously documented for African English

Evaluation Highlights

Systematic performance differentiation across model categories: open-source ASR strong on spontaneous speech but degrades on noisy non-native dialogue; multimodal LLMs more accent-robust but weaker on domain-specific named entities; proprietary models strong on clean speech but inconsistent across countries/domains
Latency-accuracy trade-off analysis showing African-English fine-tuned models achieve competitive accuracy with lower latency, and hallucination robustness measured across SOTA models remains a persistent weakness

Signal Assessment

5/10 This is a valuable, comprehensive benchmark contribution that fills a genuine and underserved evaluation gap for African-accented speech AI, but it is primarily an empirical evaluation effort rather than a new architecture, training method, or theoretical advance.

Methodology

Aggregate spontaneous and non-spontaneous speech samples from multiple existing open African-accented English datasets spanning 10+ countries and 100+ accents
Structure evaluation data into seven verticalized domains (Finance, Legal, Medical, General dialogue, Call Center, Named Entities, Hallucination Robustness) reflecting real-world application needs
Benchmark a heterogeneous model pool including open-source unimodal ASR, closed/proprietary ASR, and multimodal LLM-based speech recognition systems under consistent conditions
Perform empirical error analysis across accent, country, domain, and speech-style dimensions to surface systematic performance patterns and failure modes
Assess practical deployment factors such as latency alongside accuracy, and quantify hallucination frequency across models

System Components

Multi-domain speech corpus

Curated spontaneous and non-spontaneous speech spanning Finance, Legal, Medical, General dialogue, and Call Center verticals sourced from existing African-accented English datasets

Named Entity evaluation track

Domain-specific test set measuring ASR/LLM accuracy on recognizing named entities (e.g., proper nouns, terminology) relevant to specialized use cases

Hallucination Robustness suite

Evaluation component designed to measure the propensity of ASR and multimodal LLM systems to produce hallucinated (non-existent) transcriptions

Multi-country accent coverage

Speech samples spanning 100+ distinct African English accents across 10+ countries to test cross-regional generalization

Cross-model benchmarking framework

Standardized harness for comparing open-source, proprietary, unimodal ASR, and multimodal LLM-based speech recognition systems on identical test conditions

Results

Model Category	Strength	Weakness	Practical Implication
Open-source ASR	Excels on spontaneous speech	Degrades on noisy, non-native dialogue	Best for casual conversational use, risky for noisy/clinical settings
Multimodal LLM-based ASR	More accent-robust across countries	Struggles with domain-specific named entities	Good general-purpose fit, needs entity fine-tuning for verticals
Proprietary/closed models	High accuracy on clean speech	Significant variance by country and domain	Requires per-market validation before deployment
Fine-tuned African English models	Competitive accuracy with lower latency	Hallucinations still occur across SOTA models	Practical choice for latency-sensitive production deployment

Key Takeaways

No single ASR or multimodal LLM system dominates across all conditions—model choice should be tailored to the target domain, country, and speech style (spontaneous vs. clean/scripted)
Fine-tuning on African-accented English offers a favorable latency-accuracy trade-off, making it a strong candidate for real-time, cost-sensitive production deployments
Named entity recognition and hallucination robustness are critical blind spots for current SOTA models, warranting extra safeguards in high-stakes domains like medical and legal transcription
Multimodal LLM-based speech systems show promise for cross-accent generalization but require targeted improvement on specialized vocabulary before enterprise adoption in African markets

Abstract

Recent advances in speech-enabled AI, including Google's NotebookLM and OpenAI's speech-to-speech API, are driving widespread interest in voice interfaces globally. Despite this momentum, there exists no publicly available application-specific model evaluation that caters to Africa's linguistic diversity. We present AfriSpeech-MultiBench, the first domain-specific evaluation suite for over 100 African English accents across 10+ countries and seven application domains: Finance, Legal, Medical, General dialogue, Call Center, Named Entities and Hallucination Robustness. We benchmark a diverse range of open, closed, unimodal ASR and multimodal LLM-based speech recognition systems using both spontaneous and non-spontaneous speech conversation drawn from various open African accented English speech datasets. Our empirical analysis reveals systematic variation: open-source ASR models excels in spontaneous speech contexts but degrades on noisy, non-native dialogue; multimodal LLMs are more accent-robust yet struggle with domain-specific named entities; proprietary models deliver high accuracy on clean speech but vary significantly by country and domain. Models fine-tuned on African English achieve competitive accuracy with lower latency, a practical advantage for deployment, hallucinations still remain a big problem for most SOTA models. By releasing this comprehensive benchmark, we empower practitioners and researchers to select voice technologies suited to African use-cases, fostering inclusive voice applications for underserved communities.