Summary of Model Collapse Demystified: the Case Of Regression, by Elvis Dohmatob et al.
Model Collapse Demystified: The Case of Regression
by Elvis Dohmatob, Yunzhen Feng, Julia Kempe
First submitted to arxiv on: 12 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper studies the phenomenon of “model collapse” in large language and image generation models, where recursive training on generated data leads to performance degradation and eventual uselessness. Analytic formulae are derived for high-dimensional regression, revealing a broad range of regimes affected by this collapse. Modified scaling laws are also proposed for polynomial decaying spectral and source conditions, exhibiting new crossover phenomena. To mitigate model collapse, the authors suggest an adaptive regularization strategy based on theoretical results validated through experiments. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper explores why some AI models become less effective when they’re trained to generate more data from themselves. Imagine training a language model to generate texts, then using those generated texts to train it again, and again, and again. Eventually, the model will start producing terrible text that doesn’t make sense. This is called “model collapse”. The authors of this paper figure out why this happens and how to prevent it from happening in the first place. |
Keywords
* Artificial intelligence * Image generation * Language model * Regression * Regularization * Scaling laws
