Summary of Tilting the Odds at the Lottery: the Interplay Of Overparameterisation and Curricula in Neural Networks, by Stefano Sarao Mannelli and Yaraslau Ivashynka and Andrew Saxe and Luca Saglietti
Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks
by Stefano Sarao Mannelli, Yaraslau Ivashynka, Andrew Saxe, Luca Saglietti
First submitted to arxiv on: 3 Jun 2024
Categories
- Main: Machine Learning (stat.ML)
- Secondary: Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (cs.LG); Neurons and Cognition (q-bio.NC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper delves into the relationship between curriculum learning and overparameterization in neural networks. Building on previous works, it explores how amplifying the performance of neural networks through overparameterization can impact the effectiveness of curated example orders in guiding learners towards solving a task. The study focuses on a 2-layer network in an XOR-like Gaussian Mixture problem, revealing that high degrees of overparameterization can limit the benefits gained from curricula, providing theoretical insights into the ineffectiveness of curricula in deep learning settings. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how neural networks learn and how we can help them by giving them the right examples to practice on. It shows that making the network bigger (overparameterizing it) can actually make it harder for this helpful example ordering (curriculum) to work well. By studying a special kind of problem, they found out why curricula don’t usually work as well in deep learning. |
Keywords
» Artificial intelligence » Curriculum learning » Deep learning
