Summary of What Is Lost in Normalization? Exploring Pitfalls in Multilingual Asr Model Evaluations, by Kavya Manohar et al.
What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations
by Kavya Manohar, Leena G Pillai, Elizabeth Sherly
First submitted to arxiv on: 4 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the pitfalls in evaluating multilingual automatic speech recognition (ASR) models, particularly focusing on Indic language scripts. It examines the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta’s MMS, Seamless, and Assembly AI’s Conformer, revealing unintended consequences on performance metrics. The research demonstrates that current text normalization practices are flawed when applied to Indic scripts, artificially improving performance metrics for these languages. To address this issue, the paper proposes a shift towards developing text normalization routines that leverage native linguistic expertise. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study looks at how we evaluate multilingual speech recognition models, with a special focus on languages written in Indian script characters. It checks out what leading models like OpenAI Whisper and Meta’s MMS do to make their results comparable, but finds that this approach doesn’t work well for these languages. The model results look better than they should because of this problem. To fix this issue, the researchers suggest using native language experts to create a more accurate way to compare speech recognition models. |
