Summary of Investigating Training Strategies and Model Robustness Of Low-rank Adaptation For Language Modeling in Speech Recognition, by Yu Yu et al.
Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition
by Yu Yu, Chao-Han Huck Yang, Tuan Dinh, Sungho Ryu, Jari Kolehmainen, Roger Ren, Denis Filimonov, Prashanth G. Shivakumar, Ankur Gandhe, Ariya Rastow, Jia Xu, Ivan Bulyko, Andreas Stolcke
First submitted to arxiv on: 19 Jan 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Sound (cs.SD); Audio and Speech Processing (eess.AS)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The abstract describes research on low-rank adaptation (LoRA) with frozen pretrained language models (PLMs) to improve memory-constrained hardware performance. By introducing various LoRA training strategies, the study achieves significant word error rate reductions on two datasets: Librispeech and an internal messaging domain dataset. To evaluate the stability of LoRA-based second-pass speech recognition models, the researchers examine robustness against input perturbations using novel metrics like NPRR. The results show that advanced LoRA variants, such as dynamic rank-allocated LoRA, can alleviate performance degradation in certain perturbation scenarios, suggesting a need for comprehensive selection when using LoRA-based adaptation. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study looks at how to make language models work better on computers with limited memory. They try different ways of training the models and find that some methods work really well. They also test the models by changing small parts of the words they’re trying to recognize, and see how well they do. The results show that some new techniques can help the models stay good even when things get a little mixed up. |
Keywords
* Artificial intelligence * Lora * Low rank adaptation
