Summary of Progress Measures For Grokking on Real-world Tasks, by Satvik Golechha
Progress Measures for Grokking on Real-world Tasks
by Satvik Golechha
First submitted to arxiv on: 21 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Grokking, where machine learning models generalize well even after overfitting, is typically studied in algorithmic tasks. This paper investigates grokking in real-world datasets using deep neural networks for classification under cross-entropy loss. The authors challenge the prevailing idea that weight norms are the primary cause of grokking by showing it can occur outside expected norm ranges. To better comprehend grokking, three new progress measures are introduced: activation sparsity, absolute weight entropy, and approximate local circuit complexity. These measures are related to generalization and show a stronger correlation with grokking in real-world datasets compared to weight norms. The findings suggest that while weight norms usually correlate with grokking and proposed measures, they are not causal, and the new metrics provide insight into grokking dynamics. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper studies how machine learning models can learn to do well even after being over-trained on small datasets. Researchers have previously thought that this is because the model’s weights become “simple” or smooth, but this study shows that there are other factors at play. The authors introduce new ways to measure a model’s performance and show that these measures are better than the previous ones at predicting when a model will generalize well. This helps us understand how machine learning models work and can be used to improve their performance. |
Keywords
» Artificial intelligence » Classification » Cross entropy » Generalization » Machine learning » Overfitting
