Summary of Tas: Distilling Arbitrary Teacher and Student Via a Hybrid Assistant, by Guopeng Li et al.
TAS: Distilling Arbitrary Teacher and Student via a Hybrid Assistant
by Guopeng Li, Qiang Wang, Ke Yan, Shouhong Ding, Yuan Gao, Gui-Song Xia
First submitted to arxiv on: 16 Oct 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the potential of knowledge distillation (KD) by expanding its scope to Cross-Architecture KD (CAKD), where teachers with different architectures can transfer knowledge to a given student model. The primary challenge lies in bridging the feature gaps between heterogeneous models, which stems from their distinct inductive biases and module functions. To overcome this issue, the authors introduce an assistant model that combines features from both teacher and student modules. Furthermore, they develop a spatial-agnostic InfoNCE loss to align features after spatial smoothing, improving feature alignments in CAKD. The proposed method is evaluated on various homogeneous and heterogeneous model pairs, achieving state-of-the-art performance with a maximum gain of 11.47% on CIFAR-100 and 3.67% on ImageNet-1K. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper tries to make machines learn from each other even if they have different designs. Right now, most learning methods only work when the teacher and student models are similar. The authors want to change this by allowing teachers with different architectures to teach students. They found that there is a big gap between what these different models know, so they created an assistant model to help bridge this gap. They also developed a new way to measure how well the knowledge is transferred. The results show that their method works better than existing methods on some tasks. |
Keywords
» Artificial intelligence » Knowledge distillation » Student model
