Summary of Tracking Universal Features Through Fine-tuning and Model Merging, by Niels Horn and Desmond Elliott
Tracking Universal Features Through Fine-Tuning and Model Merging
by Niels Horn, Desmond Elliott
First submitted to arxiv on: 16 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates how model features evolve when a base Transformer language model is fine-tuned on different text domains. The study starts with a one-layer Transformer trained on a combination of the BabyLM corpus and Python code from The Stack, then adapts this base model to two new domains – TinyStories and Lua programming language. The models are merged using spherical linear interpolation. The exploration aims to provide insights into feature stability and transformation across typical transfer-learning scenarios using small-scale models and sparse auto-encoders. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how a language model changes when it’s trained on different kinds of text. They start with a simple language model, then adapt it to two new types of text – short stories and programming code in Lua. The goal is to understand what happens to the features (the underlying patterns) as the model gets fine-tuned for these new domains. |
Keywords
» Artificial intelligence » Language model » Transfer learning » Transformer
