Summary of The Joint Effect Of Task Similarity and Overparameterization on Catastrophic Forgetting — An Analytical Model, by Daniel Goldfarb et al.
The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting – An Analytical Model
by Daniel Goldfarb, Itay Evron, Nir Weinberger, Daniel Soudry, Paul Hand
First submitted to arxiv on: 23 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper examines how two factors, task similarity and overparameterization, jointly affect catastrophic forgetting in continual learning models. The researchers focus on a specific model, two-task continual linear regression, where the second task is a random orthogonal transformation of an arbitrary first task. They derive an exact analytical expression for expected forgetting and uncover a nuanced pattern, showing that highly overparameterized models exhibit the most forgetting when tasks are moderately similar. However, near the interpolation threshold, forgetting decreases monotonically with expected task similarity. The findings are validated using linear regression on synthetic data and neural networks on established permutation task benchmarks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how our brains learn new things without forgetting old skills. It’s like trying to remember a list of words, but every time you add more words, some of the original ones get forgotten. The researchers wanted to know what makes this happen, so they looked at two main factors: how similar the new tasks are to the old ones, and how much “brainpower” is being used (like having too many neurons). They found that when we have a lot of brainpower, but the new tasks are only slightly different from the old ones, we tend to forget more. But if the new tasks are very different, we remember better. This helps us understand why sometimes our brains can’t keep up with new information. |
Keywords
* Artificial intelligence * Continual learning * Linear regression * Synthetic data
