Summary of Towards Understanding the Role Of Sharpness-aware Minimization Algorithms For Out-of-distribution Generalization, by Samuel Schapiro et al.
Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalization
by Samuel Schapiro, Han Zhao
First submitted to arxiv on: 6 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the application of Sharpness-Aware Minimization (SAM) to out-of-distribution (OOD) generalization tasks. It compares eight SAM variants and finds that the original SAM outperforms the Adam baseline by 4.76% on average, while stronger SAM variants achieve an improvement of 8.01%. The paper also derives OOD generalization bounds in terms of sharpness and extends the study to gradual domain adaptation (GDA), where iterative self-training is done on intermediate domains. The results show that SAM outperforms Adam by 0.82% on average, with stronger variants achieving an improvement of 1.52%. The paper highlights a disconnection between the theoretical justification for SAM and its empirical performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how to improve machine learning models when they’re used in new situations that are different from the ones they were trained on. It tries out eight different versions of a technique called Sharpness-Aware Minimization (SAM) and finds that the original version is better than another popular method called Adam. The results show that SAM can make models work 4-8% better when used in new situations. The paper also talks about how to adapt models to slowly changing situations, where the model is trained on data from both the old and new situations. In this case, SAM works even better. |
Keywords
» Artificial intelligence » Domain adaptation » Generalization » Machine learning » Sam » Self training
