Summary of The Benefits Of Reusing Batches For Gradient Descent in Two-layer Networks: Breaking the Curse Of Information and Leap Exponents, by Yatin Dandi et al.
The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents
by Yatin Dandi, Emanuele Troiani, Luca Arnaboldi, Luca Pesce, Lenka Zdeborová, Florent Krzakala
First submitted to arxiv on: 5 Feb 2024
Categories
- Main: Machine Learning (stat.ML)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the training dynamics of two-layer neural networks learning multi-index target functions. It compares single-pass and multi-pass gradient descent, finding that multi-pass GD overcomes limitations of previous methods and efficiently learns certain types of functions in finite time. The results are based on an analysis of Dynamical Mean-Field Theory (DMFT) and provide a closed-form description of the weight projections. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how neural networks learn to recognize patterns with multiple clues. It tests different ways to train these networks and finds that one method, called multi-pass gradient descent, is better than others for learning certain types of patterns. This is important because it could help make artificial intelligence more efficient and effective. |
Keywords
* Artificial intelligence * Gradient descent
