Summary of Svft: Parameter-efficient Fine-tuning with Singular Vectors, by Vijay Lingam et al.
SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
by Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, Sujay Sanghavi
First submitted to arxiv on: 30 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach called SVFT (Sparse Vector Fine-Tuning) for parameter-efficient fine-tuning (PEFT) in pre-trained models. Unlike existing PEFT methods, such as LoRA and its variants, which freeze model weights and inject structured learnable matrices, SVFT updates the model weights by combining their singular vectors with sparse coefficients. This approach allows for fine-grained control over expressivity through the number of coefficients trained. The authors demonstrate that SVFT recovers up to 96% of full fine-tuning performance on language and vision benchmarks while training only a fraction (0.006-0.25%) of parameters, outperforming existing methods. The proposed method shows promise for efficient adaptation of pre-trained models to specific tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper introduces a new way to make computers learn faster by using less information. Existing methods try to update the computer’s knowledge by adding some extra bits, but this new approach, called SVFT, updates the computer’s memory by combining pieces of it in a special way. This allows the computer to learn more accurately while using much less information. The authors tested their method on different types of tasks and found that it performed just as well as the traditional way, but used much fewer resources. |
Keywords
* Artificial intelligence * Fine tuning * Lora * Parameter efficient
