Summary of High Dimensional Analysis Reveals Conservative Sharpening and a Stochastic Edge Of Stability, by Atish Agarwala et al.
High dimensional analysis reveals conservative sharpening and a stochastic edge of stability
by Atish Agarwala, Jeffrey Pennington
First submitted to arxiv on: 30 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Optimization and Control (math.OC); Statistics Theory (math.ST); Data Analysis, Statistics and Probability (physics.data-an)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper investigates the dynamics of the large eigenvalues of the training loss Hessian in various machine learning models and datasets. The study reveals that these eigenvalues exhibit robust features across different regimes, including an initial period of progressive sharpening followed by stabilization at a predictable value known as the edge of stability. In contrast, stochastic settings demonstrate a slower increase in eigenvalues, a phenomenon dubbed conservative sharpening. The paper presents a theoretical analysis of a high-dimensional model to explain this slowdown and identifies an alternative stochastic edge of stability that arises at small batch sizes, influenced by the trace of the Neural Tangent Kernel rather than large Hessian eigenvalues. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study explores how machine learning models behave when training on different datasets. The researchers found that some big numbers in the math behind these models stay consistent across many types of data and algorithms. They also discovered that when training with small batches of data, these big numbers change more slowly than they do when using all the data at once. This can help us understand why certain machine learning techniques work well or poorly. |
Keywords
» Artificial intelligence » Machine learning
