Summary of Discrete Multimodal Transformers with a Pretrained Large Language Model For Mixed-supervision Speech Processing, by Viet Anh Trinh et al.
Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing
by Viet Anh Trinh, Rosy Southwell, Yiwen Guan, Xinlu He, Zhiyong Wang, Jacob Whitehill
First submitted to arxiv on: 4 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents a decoder-only Discrete Multimodal Language Model (DMLM) that can be applied to multiple tasks and modalities. This model builds on recent work in discrete speech tokenization and the richness of linguistic information contained in large language models (LLMs). The authors explore various aspects of DMLM, including loss function, weight initialization, mixed training supervision, and codebook. Experimental results show that DMLM benefits from a combination of supervised and unsupervised training across multiple tasks and datasets, with notable improvements when initialized from a pretrained LLM or using a Whisper-activation-derived codebook for automatic speech recognition (ASR). |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper creates a special language model that can do many things like recognize speech, turn text into speech, and translate between different languages. It uses ideas from previous work on breaking down spoken words into individual units. The model gets better at doing these tasks by training it with lots of text data. When tested, this new model does a great job on various tasks, especially when started with information learned from another large language model. |
Keywords
» Artificial intelligence » Decoder » Language model » Large language model » Loss function » Supervised » Tokenization » Unsupervised
