Summary of Neural Collapse For Cross-entropy Class-imbalanced Learning with Unconstrained Relu Feature Model, by Hien Dang and Tho Tran and Tan Nguyen and Nhat Ho
Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model
by Hien Dang, Tho Tran, Tan Nguyen, Nhat Ho
First submitted to arxiv on: 4 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates a phenomenon known as Neural Collapse (NC) where the features in the last layer of deep neural networks collapse to their class-means during training, converging to vertices of an Equiangular Tight Frame (ETF). Recent works have employed simplified feature models to theoretically understand NC, but these studies have been limited to balanced datasets. This paper generalizes NC to imbalanced regimes under cross-entropy loss and unconstrained ReLU features, showing that the within-class features still collapse but class-means converge to orthogonal vectors with varying lengths. The study also finds that classifier weights align with scaled and centered class-means, which generalizes NC in balanced settings. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at something called Neural Collapse (NC) where deep neural networks’ features collapse to their classes during training. It’s like a pattern that happens when the network is trying to classify things. Usually, this pattern only shows up on datasets with an equal number of examples for each class. But what if the dataset has more examples from one class than others? That’s what this paper tries to figure out. They show that even in imbalanced datasets, NC still happens, but it’s a bit different. The features collapse like before, but the classes’ means move away from being perfectly equal. |
Keywords
* Artificial intelligence * Cross entropy * Relu
