Summary of Scalable Efficient Training Of Large Language Models with Low-dimensional Projected Attention, by Xingtai Lv et al.
Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
by Xingtai Lv, Ning Ding, Kaiyan Zhang, Ermo Hua, Ganqu Cui, Bowen Zhou
First submitted to arxiv on: 4 Nov 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a novel approach called Low-dimensional Projected Attention (LPA), which improves the effectiveness and efficiency of large language models (LLMs). By applying low-rank pre-training to the attention layer, LPA resolves the trade-off between performance and efficiency. The authors demonstrate the scalability and effectiveness of LPA through extensive experimentation with parameter scales ranging from 130M to 3B. Compared to vanilla Transformers, LPA achieves an approximate 5% improvement in test perplexity (ppl) and downstream task performance while saving up to 12.4% in time. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary LPA is a new way to make language models better and faster. Usually, making them faster makes them worse at understanding language. But LPA finds a way to fix this problem by only using the low-dimensional module on the attention layer. This helps both performance and efficiency. The researchers tested LPA with different numbers of parameters and found that it works well even when the model is very large or very small. |
Keywords
» Artificial intelligence » Attention » Perplexity
