Summary of Progressive Distillation Induces An Implicit Curriculum, by Abhishek Panigrahi et al.
Progressive distillation induces an implicit curriculum
by Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Andrej Risteski, Surbhi Goel
First submitted to arxiv on: 7 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores knowledge distillation, a technique where a teacher model improves the training of a student model. The authors identify an issue with this approach: a better teacher does not always yield a better student. To address this, they propose using progressive distillation, where the student learns from intermediate checkpoints of the teacher. This approach is validated through experiments on sparse parity and probabilistic context-free grammars (PCFGs), as well as real-world pre-training datasets like Wikipedia and Books. The authors find that progressive distillation accelerates the student’s learning by imparting an implicit curriculum, which is available only through intermediate checkpoints. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how to make a model learn from another better model. They found that even if you have a great teacher, it doesn’t always mean your student will be good too. So, they came up with an idea called progressive distillation, where the student learns from different stages of the teacher’s learning process. This helps the student learn faster and better than just using one final stage. The authors tested this on simple and complex tasks and found that it works well for both. |
Keywords
» Artificial intelligence » Distillation » Knowledge distillation » Student model » Teacher model
