Summary of Griffin: Mixing Gated Linear Recurrences with Local Attention For Efficient Language Models, by Soham De et al.
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
by Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, Caglar Gulcehre
First submitted to arxiv on: 29 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel recurrent neural network (RNN) architecture called Hawk is proposed, featuring gated linear recurrences that enable fast inference and scalability on long sequences. Additionally, a hybrid model called Griffin combines gated linear recurrences with local attention, demonstrating comparable performance to Llama-2 despite training on significantly fewer tokens. The models exhibit hardware efficiency during training, low latency, and high throughput during inference. Furthermore, Griffin can extrapolate on longer sequences than seen during training, outperforming Mamba on downstream tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A new type of computer model called Hawk is being developed, which is good at processing long strings of data quickly. Another model called Griffin combines two different approaches to improve its performance and ability to handle long sequences. These models can process information quickly and efficiently, making them useful for many applications. They can even work on longer sequences than they were trained on, which is helpful. |
Keywords
* Artificial intelligence * Attention * Inference * Llama * Neural network * Rnn
