Summary of Early Alignment in Two-layer Networks Training Is a Two-edged Sword, by Etienne Boursier et al.
Early alignment in two-layer networks training is a two-edged sword
by Etienne Boursier, Nicolas Flammarion
First submitted to arxiv on: 19 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the early stages of neural network training using first-order optimization methods, a crucial aspect of deep learning’s empirical success. The authors provide a quantitative description of the “early alignment phase,” which was introduced by Maennel et al. (2018). They find that small initializations lead to an alignment of neurons towards key directions, inducing sparsity in the network and influencing gradient flow at convergence. However, this alignment also leads to difficulties in minimizing the training objective, as demonstrated through a simple data example where overparameterized networks fail to converge to global minima. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how neural networks train using simple methods. It shows that when you start with small values for the network’s weights, the neurons line up in certain directions. This makes the network simpler and more sparse, which is related to why gradient descent works well for deep learning. However, this alignment can also make it harder to find the best solution, as shown by an example where overcomplicated networks get stuck in a bad place instead of finding the global minimum. |
Keywords
* Artificial intelligence * Alignment * Deep learning * Gradient descent * Neural network * Optimization
