Summary of Wasserstein Distance Rivals Kullback-leibler Divergence For Knowledge Distillation, by Jiaming Lv et al.
Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
by Jiaming Lv, Haoyuan Yang, Peihua Li
First submitted to arxiv on: 11 Dec 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Wasserstein Distance (WD) based knowledge distillation methodology addresses the limitations of traditional Kullback-Leibler Divergence (KL-Div) approaches. The logit distillation method, WKD-L, performs cross-category comparison of probabilities, leveraging interrelations among categories. Meanwhile, the feature distillation method, WKD-F, models feature distributions using a parametric approach and transfers knowledge from intermediate layers via continuous WD. The methodology is evaluated on image classification and object detection tasks, demonstrating superior performance for both logit and feature distillation approaches compared to KL-Div variants and state-of-the-art competitors. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A new way of sharing knowledge between AI models has been developed. This method uses a technique called Wasserstein Distance (WD) to compare the probabilities of different categories within an image or object. The WD approach can handle complex relationships between categories, unlike previous methods that only compared simple category-to-category probabilities. Two new techniques were introduced: logit distillation, WKD-L, and feature distillation, WKD-F. These methods outperformed existing approaches in image classification and object detection tasks. |
Keywords
» Artificial intelligence » Distillation » Image classification » Knowledge distillation » Object detection
