Summary of A Unified View Of Delta Parameter Editing in Post-trained Large-scale Models, by Qiaoyu Tang et al.
A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models
by Qiaoyu Tang, Le Yu, Bowen Yu, Hongyu Lin, Keming Lu, Yaojie Lu, Xianpei Han, Le Sun
First submitted to arxiv on: 17 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a unified framework for examining the characteristics of delta parameters in large-scale pre-trained models. Delta parameters are the differences between post-training and pre-training model parameters, and their properties have been explored through various operations like pruning, quantization, low-rank approximation, and extrapolation. The authors introduce a novel perspective based on Riemann sum approximation to explain these delta parameter editing operations. They categorize existing methods into three classes: competitive, decreased, and improved, showing how they affect model performance. Extensive experiments are conducted on visual and language models, including ViT, LLaMA 3, Qwen 2, and Mistral, confirming the theoretical findings. The authors also introduce extensions to existing techniques like DARE and BitDelta, highlighting their limitations in leveraging delta parameter properties. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps us understand how big AI models can be changed to do different tasks better. These changes are measured by something called “delta parameters.” Researchers have been trying to figure out what makes these changes work or not. The authors of this paper come up with a new way to look at these delta parameters and group the existing methods into three categories: some make the model better, some make it worse, and some keep it the same. They test their ideas on different types of models, like those for images and language. Their results show that their approach is helpful in understanding how these big AI models can be adapted to do new tasks. |
Keywords
» Artificial intelligence » Llama » Pruning » Quantization » Vit
