Summary of Training on the Edge Of Stability Is Caused by Layerwise Jacobian Alignment, By Mark Lowell and Catharine Kastner
Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignment
by Mark Lowell, Catharine Kastner
First submitted to arxiv on: 31 May 2024
Categories
- Main: Machine Learning (stat.ML)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents an innovative approach to training neural networks, addressing the issue of the sharpness of the Hessian matrix during training. By using an exponential Euler solver, the authors successfully model the underlying dynamical system defined by the gradient flow of the training loss, avoiding the edge of stability that traditional methods encounter. The work demonstrates that the increase in Hessian sharpness is caused by layerwise Jacobian matrices becoming aligned, leading to a power-law scaling of alignment with dataset size. This research has significant implications for understanding and improving neural network training dynamics. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary During neural network training, the sharpness of the Hessian matrix increases until the edge of stability is reached. To avoid this, an exponential Euler solver is used to accurately model the true gradient descent dynamics. The study shows that layerwise Jacobian matrices becoming aligned causes the increase in Hessian sharpness, and this alignment scales with dataset size. |
Keywords
» Artificial intelligence » Alignment » Gradient descent » Neural network
