Summary of On Difficulties Of Attention Factorization Through Shared Memory, by Uladzislau Yorsh et al.
On Difficulties of Attention Factorization through Shared Memory
by Uladzislau Yorsh, Martin Holeňa, Ondřej Bojar, David Herel
First submitted to arxiv on: 31 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Transformers have transformed deep learning across various fields like natural language processing, computer vision, and audio processing. Their strength lies in their attention mechanism, which allows for discovering complex relationships between inputs. However, this mechanism’s quadratic time and memory complexity pose challenges for larger inputs. Researchers are investigating models like Linear Unified Nested Attention (Luna) or Memory Augmented Transformer, leveraging external learnable memory to reduce attention computation complexity down to linear or propagate information between chunks in chunk-wise processing. Our findings challenge conventional thinking on these models, revealing that interfacing with the memory directly through an attention operation is suboptimal and that performance can be significantly improved by filtering the input signal before communicating with memory. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Transformers have changed the game for deep learning across many areas like language, vision, and audio. They’re great at finding connections between things. But this ability comes with a big cost – it’s slow and uses a lot of memory. To solve this problem, researchers are creating new models that use extra learnable “memory” to make calculations faster or to share information in chunks. Our research shows that these models don’t work as well when they try to directly talk to the memory using attention. Instead, we found that by cleaning up the input signal before talking to the memory, we can get much better results. |
Keywords
* Artificial intelligence * Attention * Deep learning * Natural language processing * Transformer
