Summary of Grokfast: Accelerated Grokking by Amplifying Slow Gradients, By Jaerin Lee et al.
Grokfast: Accelerated Grokking by Amplifying Slow Gradients
by Jaerin Lee, Bong Gyun Kang, Kihoon Kim, Kyoung Mu Lee
First submitted to arxiv on: 30 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the phenomenon of grokking, where a model achieves delayed generalization after overfitting to training data. The authors focus on accelerating this process for machine learning practitioners. By decomposing parameter trajectories into fast-varying and slow-varying components, they develop an algorithm that amplifies the slow-varying gradients to accelerate grokking by over 50 times with minimal code changes. The approach is shown to be effective across diverse tasks involving images, languages, and graphs, making practical applications possible. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Grokking is a mysterious phenomenon in machine learning where a model generalizes well long after being trained on data. Researchers are trying to understand why this happens and how to make it happen faster. The authors of this paper took a closer look at what’s happening when a model overfits and then suddenly becomes good at recognizing patterns. They found that by looking at the way the model changes over time, they can speed up this process by a lot – 50 times or more! This could be very useful for people who work with machine learning models. The new technique works well on lots of different types of data, including pictures, words, and graphs. |
Keywords
* Artificial intelligence * Generalization * Machine learning * Overfitting
