Summary of Switch Ema: a Free Lunch For Better Flatness and Sharpness, by Siyuan Li et al.
Switch EMA: A Free Lunch for Better Flatness and Sharpness
by Siyuan Li, Zicheng Liu, Juanxi Tian, Ge Wang, Zedong Wang, Weiyang Jin, Di Wu, Cheng Tan, Tao Lin, Yang Liu, Baigui Sun, Stan Z. Li
First submitted to arxiv on: 14 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a novel approach to deep neural network (DNN) optimization, building upon the widely used Exponential Moving Average (EMA) regularization technique. By modifying EMA to switch its parameters to the original model after each epoch, dubbed Switch EMA (SEMA), the authors demonstrate that SEMA can improve DNN generalization performance and convergence speed without additional computational cost. Theoretical and empirical evaluations show that SEMA achieves better trade-offs between flatness and sharpness in optimization landscapes, outperforming existing WA methods on a range of discriminative, generative, and regression tasks across vision and language datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps deep learning models work better without using extra computer power. It shows how to improve the performance and speed of training these models by tweaking a popular technique called Exponential Moving Average (EMA). The authors tested their new approach on many different tasks and datasets, and it worked well in all cases. |
Keywords
* Artificial intelligence * Deep learning * Generalization * Neural network * Optimization * Regression * Regularization
