Summary of Exploring Grokking: Experimental and Mechanistic Investigations, by Hu Qiye et al.
Exploring Grokking: Experimental and Mechanistic Investigations
by Hu Qiye, Zhou Hao, Yu RuoXi
First submitted to arxiv on: 14 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper delves into the phenomenon of “grokking” in over-parameterized neural networks, where initial memorization of training data is followed by a sharp transition to perfect generalization. The authors conduct extensive experiments to understand this behavior, examining factors such as training data fraction, model architecture, and optimization methods. They also explore various research perspectives on the underlying mechanism of grokking. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Grokking in neural networks means that they initially remember all training data perfectly but then suddenly start making good predictions on new data too. Researchers are curious about why this happens. This paper looks at the phenomenon through many experiments and explores what others have found out about it so far. They want to understand how things like the amount of training data, the type of neural network, and how we train them affect grokking. |
Keywords
* Artificial intelligence * Generalization * Neural network * Optimization
