Summary of When Will Gradient Regularization Be Harmful?, by Yang Zhao et al.
When Will Gradient Regularization Be Harmful?
by Yang Zhao, Hao Zhang, Xiuyuan Hu
First submitted to arxiv on: 14 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the effectiveness of gradient regularization (GR) in training over-parameterized deep neural networks. GR has shown promising results, but its limitations are not well understood. The authors reveal that GR can cause performance degeneration in adaptive optimization scenarios, particularly with learning rate warmup. They propose three GR warmup strategies to relax the regularization effect during the initial training stage and ensure stable gradient accumulation. Experiments on Vision Transformer models confirm the effectiveness of these strategies, improving model performance by up to 3% on Cifar10 compared to baseline GR. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how well a technique called gradient regularization works in training big neural networks. Gradient regularization helps the network learn better, but it can also make things worse if not used correctly. The authors found that this problem happens more often when the network is adapting quickly and then slows down. They came up with three new ways to use gradient regularization that work better and can even make the network 3% better on a specific task. |
Keywords
* Artificial intelligence * Optimization * Regularization * Vision transformer
