Summary of Momentum-sam: Sharpness Aware Minimization Without Computational Overhead, by Marlon Becker et al.
Momentum-SAM: Sharpness Aware Minimization without Computational Overhead
by Marlon Becker, Frederick Altrock, Benjamin Risse
First submitted to arxiv on: 22 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A new optimization algorithm for deep neural networks called Momentum-SAM (MSAM) is proposed, which addresses the limitations of its predecessor, Sharpness Aware Minimization (SAM). MSAM modifies SAM by perturbing parameters in the direction of the accumulated momentum vector, achieving low sharpness without significant computational overhead or memory demands. This approach outperforms traditional stochastic gradient descent (SGD) and Adam optimizers while reducing overfitting. The paper investigates the separable mechanisms underlying NAG, SAM, and MSAM regarding training optimization and generalization. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary MSAM is a new way to train deep neural networks that helps them generalize better without using too many computations. It’s like a shortcut that makes the training process more efficient. The idea is based on another algorithm called Nesterov Accelerated Gradient (NAG) that also tries to find the right direction for the optimization. MSAM takes this idea and combines it with SAM, which helps to avoid overfitting. This new approach works better than some other popular methods like SGD and Adam. |
Keywords
* Artificial intelligence * Generalization * Optimization * Overfitting * Sam * Stochastic gradient descent
