Summary of Robustness to Distribution Shifts Of Compressed Networks For Edge Devices, by Lulan Shen et al.
Robustness to distribution shifts of compressed networks for edge devices
by Lulan Shen, Ali Edalati, Brett Meyer, Warren Gross, James J. Clark
First submitted to arxiv on: 22 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Compressed deep neural networks (DNNs) are crucial for edge devices with limited computation resources. However, these compressed networks often encounter new tasks in a target domain, which differs from the source domain where they were trained. This study investigates the robustness of compressed DNNs to data distribution shifts and adversarial perturbations. The findings reveal that compressed models are less robust to domain shifts than their original counterparts. Interestingly, larger networks are more vulnerable to losing robustness even when compressed to a similar size as smaller ones. Compact networks obtained through knowledge distillation show significant robustness gains compared to pruned networks. Finally, post-training quantization emerges as a reliable method for achieving robustness against distribution shifts, outperforming both pruned and distilled models in terms of robustness. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Compressed deep learning models are important for devices with limited power. These models often need to do new tasks in places they weren’t trained. This study looks at how well these compressed models work when the data is different from what they were trained on. The results show that compressed models don’t handle changes as well as their original versions. Surprisingly, bigger networks are more likely to lose their ability to handle changes even if they’re made smaller. Models that learn from each other (knowledge distillation) do a better job of handling changes than those that simply remove parts (pruning). Finally, making models work on devices with limited power helps them handle changes better. |
Keywords
* Artificial intelligence * Deep learning * Knowledge distillation * Pruning * Quantization
