Summary of Robust Knowledge Distillation Based on Feature Variance Against Backdoored Teacher Model, by Jinyin Chen et al.
Robust Knowledge Distillation Based on Feature Variance Against Backdoored Teacher Model
by Jinyin Chen, Xiaoming Zhao, Haibin Zheng, Xiao Li, Sheng Xiang, Haifeng Guo
First submitted to arxiv on: 1 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a robust knowledge distillation method called RobustKD that compresses deep neural networks while mitigating backdoors. The authors note that current KD methods focus on improving the student model’s performance without considering robustness against backdoor attacks. They demonstrate that even well-trained teacher models can contain backdoors, which are transferred to the student model during the distillation process. RobustKD addresses this issue by compressing the model while reducing feature variance between the teacher and student models. The proposed method achieves comparable main task performance to the teacher model, robustness against backdoor attacks, and generic applicability across different datasets and DNN architectures. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary RobustKD is a new way to teach machines how to learn without mistakes. Normally, when we train a machine learning model on lots of data, it can pick up bad habits from that data. This paper shows how to fix this problem by teaching the model new ways to work that are better and more reliable. The method uses something called “knowledge distillation” which is like training a student model based on an expert model. By making sure the student model learns in a way that is different from the bad habits in the data, we can make it less likely to get things wrong. |
Keywords
» Artificial intelligence » Distillation » Knowledge distillation » Machine learning » Student model » Teacher model
