Summary of Parameter Efficient Quasi-orthogonal Fine-tuning Via Givens Rotation, by Xinyu Ma et al.
Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation
by Xinyu Ma, Xu Chu, Zhibang Yang, Yang Lin, Xin Gao, Junfeng Zhao
First submitted to arxiv on: 5 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper addresses the issue of promoting parameter efficiency in fine-tuning large-scale pre-trained language models for various downstream tasks, while preserving their knowledge. The authors propose quasi-Givens Orthogonal Fine-Tuning (qGOFT), which reduces the complexity from O(d^2) to O(d) by using Givens rotations and introduces flexible norm and relative angular adjustments under soft orthogonality regularization. The method is shown to be effective in various tasks and pre-trained models, improving both efficiency and adaptation capabilities. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper talks about making computers better at learning from big models they were trained on. They want to make it more efficient so the computer can learn faster. To do this, they use a new way of adjusting the model’s parameters that is really good at keeping what the original model learned and also being able to adapt to new tasks. This means the computer will be able to learn from big models even better and faster. |
Keywords
* Artificial intelligence * Fine tuning * Regularization
