Summary of Deep Learning Through a Telescoping Lens: a Simple Model Provides Empirical Insights on Grokking, Gradient Boosting & Beyond, by Alan Jeffares et al.
Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond
by Alan Jeffares, Alicia Curth, Mihaela van der Schaar
First submitted to arxiv on: 31 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the surprising behaviors of deep learning models by developing a simple yet accurate model of a trained neural network. The proposed model, consisting of a sequence of first-order approximations, is used to derive new empirical insights on various prominent phenomena in the literature, including double descent, grokking, linear mode connectivity, and tabular data challenges. Across three case studies, the authors demonstrate how this model can be applied to predict and understand the unexpected performance of neural networks. The paper also presents a pedagogical formalism that allows researchers to isolate components of the training process, providing insights into design choices such as architecture and optimization strategy. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research tries to figure out why deep learning sometimes works in strange ways. To do this, they created a simple model that shows how a neural network is trained. They tested their model on three different areas of study and found it could help understand some weird phenomena that happen with neural networks. This new tool can also be used to see the effects of design choices when training a neural network. |
Keywords
» Artificial intelligence » Deep learning » Neural network » Optimization
