Summary of Unified View Of Grokking, Double Descent and Emergent Abilities: a Perspective From Circuits Competition, by Yufei Huang et al.
Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition
by Yufei Huang, Shengding Hu, Xu Han, Zhiyuan Liu, Maosong Sun
First submitted to arxiv on: 23 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a unified framework for understanding various phenomena in deep learning, including grokking, double descent, and emergent abilities in large language models. The framework focuses on the competition between memorization and generalization circuits, initially employed to explain grokking but extended to cover model sizes and training data volumes. It delineates four distinct training dynamics depending on varying combinations of model size and training data quantity. Experimental results support two predictions regarding double descent, while also demonstrating how algorithm tasks can be turned into emergent abilities in the multi-task learning paradigm. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper explains some weird things that happen with deep learning models, like when they suddenly get very good at something or start doing unexpected things. Researchers have been trying to understand these phenomena, which are important for making better AI models. This paper proposes a framework that helps us make sense of three specific things: grokking, double descent, and emergent abilities in language models. It shows how different combinations of model size and training data can affect how well the model does, and provides some predictions about when these phenomena will happen. |
Keywords
* Artificial intelligence * Deep learning * Generalization * Multi task
