Summary of Multi-matrix Factorization Attention, by Jingcheng Hu et al.
Multi-matrix Factorization Attention
by Jingcheng Hu, Houyi Li, Yinmin Zhang, Zili Wang, Shuigeng Zhou, Xiangyu Zhang, Heung-Yeung Shum, Daxin Jiang
First submitted to arxiv on: 26 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes novel attention architectures, Multi-matrix Factorization Attention (MFA) and MFA-Key-Reuse (MFA-KR), designed to maintain strong performance under stringent Key-Value cache constraints. The proposed methods enhance model capacity by scaling up attention heads through low-rank matrix factorization in the Query-Key circuit, reducing memory requirements while achieving comparable performance to Multi-Head Attention (MHA) and surpassing existing variants like MLA. The architectures are evaluated on large-scale experiments, demonstrating reduced Key-Value cache usage by up to 56% and 93.7%, respectively. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper introduces new attention models that work well even when memory is limited. It proposes two ways to improve the way attention works: MFA, which makes more efficient use of information, and MFA-KR, which reduces memory usage even further. These models are tested on big datasets and perform as well as or better than existing methods, while using less memory. |
Keywords
» Artificial intelligence » Attention » Multi head attention
