Summary of What Is Left After Distillation? How Knowledge Transfer Impacts Fairness and Bias, by Aida Mohammadshahi et al.
What is Left After Distillation? How Knowledge Transfer Impacts Fairness and Bias
by Aida Mohammadshahi, Yani Ioannou
First submitted to arxiv on: 10 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computers and Society (cs.CY); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the impact of Knowledge Distillation, a popular method for compressing Deep Neural Networks, on class-wise accuracy. It shows that distillation can affect up to 41% of classes in balanced image classification datasets like CIFAR-100 and ImageNet, leading to changes in class bias. The study also evaluates fairness metrics like Demographic Parity Difference (DPD) and Equalized Odds Difference (EOD) on models trained with CelebA, Trifeature, and HateXplain datasets. Results suggest that increasing the distillation temperature improves the distilled student model’s fairness, surpassing the teacher model at high temperatures. This study highlights the potential effects of Knowledge Distillation on class bias and emphasizes caution when using distilled models in sensitive application domains. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Knowledge Distillation is a way to make Deep Neural Networks smaller. Usually, it works well and keeps performance good. But research shows that distillation can actually change how different classes are classified, which might not be bad or good – it just depends on the context. The study looked at fairness metrics like Demographic Parity Difference (DPD) and Equalized Odds Difference (EOD) to see if distillation makes models fairer. It found that making the distillation temperature higher makes the student model more fair, even better than the teacher model! This means we need to be careful when using these compressed models for important tasks. |
Keywords
» Artificial intelligence » Distillation » Image classification » Knowledge distillation » Student model » Teacher model » Temperature
