Summary of Visualising Feature Learning in Deep Neural Networks by Diagonalizing the Forward Feature Map, By Yoonsoo Nam et al.
Visualising Feature Learning in Deep Neural Networks by Diagonalizing the Forward Feature Map
by Yoonsoo Nam, Chris Mingard, Seok Hyeong Lee, Soufiane Hayou, Ard Louis
First submitted to arxiv on: 5 Oct 2024
Categories
- Main: Machine Learning (stat.ML)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Deep learning models, like deep neural networks (DNNs), excel at automatically learning data representations without human input. This paper presents an approach for analyzing how DNNs learn features by decomposing them into two components: a feature-map that maps the input to the post-activation of the penultimate layer and a final linear layer for classification. By diagonalizing the feature-map with respect to the gradient descent operator, researchers can track feature learning by monitoring the eigenfunctions and eigenvalues’ changes during training. Across various architectures and datasets, DNNs converge to either a minimal feature (MF) regime or an extended feature (EF) regime, depending on the specific model and data combination. The MF regime is characterized by using only a few features equal to the number of classes, resembling the neural collapse phenomenon studied at longer training times. Optimal generalization performance typically coincides with the MF regime, but poor performance can also occur within it. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary DNNs are super smart! They can learn about data without humans telling them what to do. This paper helps us understand how DNNs learn by breaking them into two parts: a map that changes input data and a final check to see if the data is correct or not. By looking at how this map changes during learning, we can see which features are important. Most of the time, DNNs use only a few important features to make good predictions. Sometimes, they might use too many features and make mistakes. |
Keywords
» Artificial intelligence » Classification » Deep learning » Feature map » Generalization » Gradient descent
