Summary of On the Emergence Of Cross-task Linearity in the Pretraining-finetuning Paradigm, by Zhanpeng Zhou et al.
On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm
by Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, Junchi Yan
First submitted to arxiv on: 6 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper uncovers a fascinating linear phenomenon in deep learning models that are initialized from a common pretrained checkpoint and fine-tuned on different tasks. Specifically, when interpolating model weights between two fine-tuned models, features at each layer exhibit approximately equal linear interpolation. The authors provide comprehensive empirical evidence supporting this Cross-Task Linearity (CTL) for finetuned models starting from the same pretrained checkpoint. They propose that neural networks function as linear maps, mapping parameters to features. This insight sheds new light on model merging/editing by translating operations between parameter and feature spaces. Furthermore, the paper investigates the root cause of CTL, highlighting the role of pretraining in deep learning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The study finds a surprising pattern in how AI models work. When you take two different models that were both trained from the same starting point, and then “mix” their weights to create a new model, something cool happens: the features (or building blocks) of the new model are just as mixed as the original models! The researchers show that this pattern, called Cross-Task Linearity, happens consistently when using the same starting point for both models. They think this might be because AI networks can be thought of as simple “map-making” tools, taking in parameters and producing features. This new understanding could help us better merge or edit AI models in the future. |
Keywords
* Artificial intelligence * Deep learning * Pretraining
