Summary of Linguistic Collapse: Neural Collapse in (large) Language Models, by Robert Wu and Vardan Papyan
Linguistic Collapse: Neural Collapse in (Large) Language Models
by Robert Wu, Vardan Papyan
First submitted to arxiv on: 28 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the phenomenon of neural collapse (NC) in language modelling, specifically in causal language models (CLMs). NC is characterized by top-layer representations collapsing into their class means, which become equinorm, equiangular and aligned with the classifiers. The authors explore how scaling architectures and training affects the progression towards NC in CLMs. They find that NC properties developing with scale (and regularization) are linked to generalization, and there is evidence of some relationship between NC and generalization independent of scale. The study underscores the generality of NC as it extends to the novel setting of language modelling. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at a phenomenon called neural collapse in language models. Neural collapse happens when the top layers of a model’s representation become equal, aligned with each other and with the classifier. This can help language models be more generalizable and robust. The authors investigate how making the model bigger or training it for longer affects this phenomenon. They find that as the model gets bigger and is trained better, it becomes more likely to exhibit neural collapse, which in turn helps it generalize better. |
Keywords
* Artificial intelligence * Generalization * Regularization
