Summary of Exact, Tractable Gauss-newton Optimization in Deep Reversible Architectures Reveal Poor Generalization, by Davide Buffelli et al.
Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization
by Davide Buffelli, Jamie McGowan, Wangkun Xu, Alexandru Cioba, Da-shan Shiu, Guillaume Hennequin, Alberto Bernacchia
First submitted to arxiv on: 12 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the effectiveness of second-order optimization methods, specifically Gauss-Newton updates, for training deep neural networks. While these methods can accelerate training and produce better performance, their generalization properties are still unclear due to the difficulties in carrying out theoretical investigations outside simplified model classes. The authors address this gap by showing that exact GN updates take a tractable form in certain reversible architectures, allowing them to study the training and generalization properties of the GN optimizer. They find that exact GN generalizes poorly, leading to rapid saturation on the training loss and overfitting of each mini-batch. This poor generalization is attributed to the network’s “lazy” regime, where the neural tangent kernel changes little during training. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper explores how second-order optimization methods can help train deep neural networks faster and better. Researchers have been trying to understand if these methods are good for all types of problems or just some. The authors found a way to make exact second-order updates work in certain types of networks, which lets them study how well this method performs. They discovered that when they used the method, it didn’t generalize well and actually got worse over time. |
Keywords
» Artificial intelligence » Generalization » Optimization » Overfitting
