Summary of Speech Reallm — Real-time Streaming Speech Recognition with Multimodal Llms by Teaching the Flow Of Time, By Frank Seide et al.
Speech ReaLLM – Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time
by Frank Seide, Morrie Doulaty, Yangyang Shi, Yashesh Gaur, Junteng Jia, Chunyang Wu
First submitted to arxiv on: 13 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Sound (cs.SD); Audio and Speech Processing (eess.AS)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces Speech ReaLLM, a new Automatic Speech Recognition (ASR) architecture that combines “decoder-only” ASR with Recurrent Neural Transducer (RNN-T) to enable real-time streaming of multimodal Large Language Models (LLMs). This is the first “decoder-only” ASR architecture designed for continuous audio processing without explicit end-pointing. The authors also introduce the more general ReaLLM (“real-time LLM”) approach, which generates output after every input token received in real-time. The paper reports WERs of 3.0% and 7.4% on Librispeech “test” for an 80M Speech ReaLLM without an external LM or auxiliary loss, only slightly above a larger Attention-Encoder-Decoder baseline. The authors also demonstrate that LLM architectures can learn to represent and reproduce the flow of time. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper creates a new way for computers to recognize and understand spoken words in real-time. It combines two existing technologies, RNN-T and ASR, to make it possible for Large Language Models (LLMs) to process continuous audio without stopping. The authors tested their approach on a big dataset called Librispeech and found that it works well, with an error rate of 3-7%. This is an important step towards making computers better at understanding spoken language in real-time. |
Keywords
» Artificial intelligence » Attention » Decoder » Encoder decoder » Rnn » Token
