Summary of The Impact Of Geometric Complexity on Neural Collapse in Transfer Learning, by Michael Munn et al.
The Impact of Geometric Complexity on Neural Collapse in Transfer Learning
by Michael Munn, Benoit Dherin, Javier Gonzalvo
First submitted to arxiv on: 24 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Many large-scale computer vision and language models owe their success to transfer learning via pre-training foundation models. However, the underlying theoretical framework remains incomplete. To bridge this gap, researchers have introduced metrics like flatness of the loss surface and neural collapse to understand implicit biases during pre-training. This paper delves into the geometric complexity of learned representations as a key mechanism linking these two concepts. Experiments and theory demonstrate how mechanisms affecting geometric complexity influence neural collapse, with implications for better performance on downstream tasks, particularly in few-shot settings. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large-scale computer vision and language models are very good at doing certain tasks. They can even learn from one task and use that knowledge to do other tasks well too. But scientists didn’t have a clear explanation for why this was happening. They came up with some new ways to measure how well these models were pre-trained, like flatness of the loss surface and neural collapse. This paper explores the idea that the way these models learn representations is important for understanding how they transfer knowledge to other tasks. The research shows that making changes to how models learn representations can actually help them do better on new tasks, especially when only given a few examples. |
Keywords
» Artificial intelligence » Few shot » Transfer learning
