Summary of Initialization Matters: on the Benign Overfitting Of Two-layer Relu Cnn with Fully Trainable Layers, by Shuning Shang et al.
Initialization Matters: On the Benign Overfitting of Two-Layer ReLU CNN with Fully Trainable Layers
by Shuning Shang, Xuran Meng, Yuan Cao, Difan Zou
First submitted to arxiv on: 24 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed research extends the analysis of benign overfitting to two-layer ReLU convolutional neural networks (CNNs) with fully trainable layers, aiming to bridge the gap between theoretical and practical applications. The study reveals that initialization scaling plays a crucial role in training dynamics, where large scales lead to rapid growth of the hidden layer and minimal changes in the output layer, whereas small scales result in more complex interactions and joint growth. Additionally, the research provides nearly matching upper and lower bounds on test errors, highlighting the conditions for achieving or avoiding benign overfitting. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Benign overfitting is when neural networks fit training data perfectly but still work well with new information. Researchers have studied this phenomenon a lot, but most of their work has been limited to simple networks. This paper looks at more complex networks called convolutional neural networks (CNNs) that are closer to what’s used in real-life applications. The study finds that how the network is initialized (started off) matters a lot. If it starts with large weights, the hidden layer grows quickly while the output layer doesn’t change much. But if it starts with small weights, both layers work together and grow at the same rate. The paper also provides some rules for when this benign overfitting happens or not. |
Keywords
» Artificial intelligence » Overfitting » Relu
