Summary of Pissa: Principal Singular Values and Singular Vectors Adaptation Of Large Language Models, by Fanxu Meng et al.
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
by Fanxu Meng, Zhaohui Wang, Muhan Zhang
First submitted to arxiv on: 3 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces Principal Singular values and Singular vectors Adaptation (PiSSA), a method that fine-tunes large language models (LLMs) by updating the principal components of the original model while freezing the residual parts. This approach enables faster convergence and improved performance compared to LoRA, which updates the entire adapter matrix. PiSSA is demonstrated to outperform LoRA on 12 different models across 5 NLG and 8 NLU tasks, achieving an accuracy of 72.86% on the GSM8K benchmark. Additionally, PiSSA is compatible with quantization, allowing for further memory reduction during fine-tuning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary PiSSA is a new way to make language models better by updating only the most important parts. This helps them learn faster and do better on tasks like language translation and question-answering. The idea is to use the original model as a starting point, then update just the parts that are most important for the task at hand. This makes it faster and more efficient than other methods. |
Keywords
* Artificial intelligence * Fine tuning * Lora * Quantization * Question answering * Translation
