Summary of Pmss: Pretrained Matrices Skeleton Selection For Llm Fine-tuning, by Qibin Wang et al.
PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
by Qibin Wang, Xiaolin Hu, Weikai Xu, Wei Liu, Jian Luan, Bin Wang
First submitted to arxiv on: 25 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes PMSS (Pre-trained Matrices Skeleton Selection), a novel method for low-rank adaptation that enables high-rank updates with low computational costs. Unlike traditional LoRA methods, which suffer from limitations in their low-rank assumption and suboptimal initialization methods, PMSS leverages pre-trained weights to select skeletons and learn only small matrices. The proposed approach outperforms LoRA and other fine-tuning methods across various tasks, such as the DROP benchmark and math reasoning, with significantly fewer trainable parameters. Notably, PMSS achieves impressive gains of +3.4%/+5.9% on LLaMA2-7B/13B, +12.89%/+5.61%/+3.11% on LLaMA2-7B, Mistral-7B, and Gemma-7B of GSM8K. The code and model will be released soon. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper introduces a new way to improve a type of machine learning called low-rank adaptation. Right now, these methods have some limitations that make them less efficient. To fix this, the researchers created something called PMSS (Pre-trained Matrices Skeleton Selection). It works by taking advantage of pre-trained knowledge and only updating small parts of the model. This makes it much faster and more accurate than other methods. In tests, PMSS performed better on complex tasks like understanding math problems and reading comprehension. The team plans to release their code and model soon. |
Keywords
» Artificial intelligence » Fine tuning » Lora » Low rank adaptation » Machine learning
