Summary of Why Does Knowledge Distillation Work? Rethink Its Attention and Fidelity Mechanism, by Chenqi Guo et al.
Why does Knowledge Distillation Work? Rethink its Attention and Fidelity Mechanism
by Chenqi Guo, Shiwei Zhong, Xiaofeng Liu, Qianli Feng, Yinglong Ma
First submitted to arxiv on: 30 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV); Image and Video Processing (eess.IV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates whether Knowledge Distillation (KD) really works as a knowledge transfer procedure. It challenges the conventional wisdom that a perfect mimicry of the student to its teacher is desired. Instead, it suggests that diverse attentions in teachers contribute to better student generalization at the expense of reduced fidelity in ensemble KD setups. The authors use data augmentation strengths to increase diversity and reduce mutual information between teachers and students, leading to improved generalization performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary KD aims to transfer knowledge from a teacher model to a student model. However, research has shown that this approach doesn’t always improve student generalization. This paper explores the reasons behind this phenomenon and proposes a new perspective on optimizing student model performance. It suggests that increasing data augmentation strengths can lead to better generalization by reducing mutual information between teachers and students. |
Keywords
» Artificial intelligence » Data augmentation » Generalization » Knowledge distillation » Student model » Teacher model
