Summary of Corda: Context-oriented Decomposition Adaptation Of Large Language Models For Task-aware Parameter-efficient Fine-tuning, by Yibo Yang et al.
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
by Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, Bernard Ghanem
First submitted to arxiv on: 7 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes CorDA, a novel approach for parameter-efficient fine-tuning (PEFT) in language models. Unlike existing methods, which build adapters agnostically of the context, CorDA decomposes linear layers using singular value decomposition (SVD) oriented by the task or world knowledge to maintain. This enables two adaptations: knowledge-preserved and instruction-previewed. The former initializes a learnable adapter with frozen components that preserve pre-trained world knowledge. The latter trains an adapter using instructions from the fine-tuning task, such as math or coding. Extensive experiments on Math, Code, and Instruction Following tasks demonstrate the effectiveness of CorDA. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper introduces a new way to make language models better at doing specific tasks without forgetting what they already know. They do this by breaking down the model into smaller pieces that can be adjusted separately based on the task. This helps the model remember important information while still learning new things. The authors test their approach on three different types of tasks and show that it works well. |
Keywords
» Artificial intelligence » Fine tuning » Parameter efficient
