Summary of Grokking at the Edge Of Linear Separability, by Alon Beck et al.
Grokking at the Edge of Linear Separability
by Alon Beck, Noam Levi, Yohai Bar-Sinai
First submitted to arxiv on: 6 Oct 2024
Categories
- Main: Machine Learning (stat.ML)
- Secondary: Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (cs.LG); Mathematical Physics (math-ph)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper studies the generalization properties of binary logistic classification in a simplified setting, where a “memorizing” and “generalizing” solution can always be strictly defined. The authors analyze the asymptotic long-time dynamics of logistic classification on a random feature model with a constant label and show that it exhibits Grokking, characterized by delayed generalization and non-monotonic test loss. They find that Grokking is amplified when classification is applied to training sets which are on the verge of linear separability. The paper also proves the implicit bias of the logistic loss will cause the model to overfit if the training data is linearly separable from the origin. Additionally, the authors examine a tractable one-dimensional toy model that quantitatively captures the key features of the full model. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how well machine learning models can generalize and memorize in a simple setting. The researchers study how logistic regression works on random data with a constant label and find that it shows something called Grokking, which is when the model takes time to learn from new data. They also find that if the training data is almost separable, the model might overfit for a long time before learning. This behavior is similar to what happens in some physical systems. |
Keywords
» Artificial intelligence » Classification » Generalization » Logistic regression » Machine learning
