Summary of The Persistence Of Neural Collapse Despite Low-rank Bias: An Analytic Perspective Through Unconstrained Features, by Connall Garrod and Jonathan P. Keating
The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features
by Connall Garrod, Jonathan P. Keating
First submitted to arxiv on: 30 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the phenomenon of “neural collapse” in deep neural networks, where their final layer features and weights exhibit a simple structure. The authors observe that this phenomenon is not unique to the final layer, but also extends to layers beyond. They find that this structure is generally suboptimal in expressive networks due to a low-rank bias induced by regularization. To understand this bias better, the authors analyze its influence on various solutions and explore how it induces specific structures in the singular values of the weights at global optima. Additionally, they examine the loss surface of these models, providing evidence that the frequent observation of deep neural collapse in practice may be due to its higher degeneracy on the loss surface. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about why some deep learning models have a special kind of simplicity in their features and weights. They found that this isn’t just true for the final layer, but also happens in other layers too. The problem is that this simplicity isn’t actually the best way to make these models work well. Instead, it’s because of how we’re training them, which makes them favor simple solutions over more complex ones. The authors looked into why this happens and found some interesting things about the shapes of the model’s weights and the loss function. |
Keywords
» Artificial intelligence » Deep learning » Loss function » Regularization
