Summary of Early Directional Convergence in Deep Homogeneous Neural Networks For Small Initializations, by Akshay Kumar and Jarvis Haupt
Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations
by Akshay Kumar, Jarvis Haupt
First submitted to arxiv on: 12 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Optimization and Control (math.OC); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Deep homogeneous neural networks are studied in this research paper, focusing on their gradient flow dynamics when training is done. The study assumes the networks have locally Lipschitz gradients and an order of homogeneity strictly greater than two. It’s found that during early stages of training, weights remain small in norm and approximately converge to Karush-Kuhn-Tucker (KKT) points of a recently introduced neural correlation function. Additionally, the KKT points are studied for feed-forward networks with Leaky ReLU and polynomial Leaky ReLU activations, deriving necessary and sufficient conditions for rank-one KKT points. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how deep neural networks behave when they’re being trained. It shows that early on in training, the weights stay small and point towards special points called Karush-Kuhn-Tucker (KKT) points. The researchers also studied what makes these special points happen for different types of networks. |
Keywords
* Artificial intelligence * Relu
