Summary of Pruning by Explaining Revisited: Optimizing Attribution Methods to Prune Cnns and Transformers, By Sayed Mohammad Vakilzadeh Hatefi et al.
Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformers
by Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Reduan Achtibat, Thomas Wiegand, Wojciech Samek, Sebastian Lapuschkin
First submitted to arxiv on: 22 Aug 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Aiming to optimize Deep Neural Networks (DNNs) for efficiency and reduced computational costs, researchers have turned to pruning unnecessary components. Previous work leveraged attribution methods from eXplainable AI (XAI) to prune DNNs in a few-shot manner. Building upon this concept, we propose explicitly optimizing hyperparameters of attribution methods for the task of pruning. This approach yields higher model compression rates for large transformer- and convolutional architectures like VGG, ResNet, and ViT on ImageNet classification tasks while maintaining high performance. Notably, our experiments reveal that transformers exhibit a higher degree of over-parameterization compared to convolutional neural networks. Our PyTorch implementation is publicly available at https://github.com/erfanhatefi/Pruning-by-eXplaining-in-PyTorch. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine trying to solve really complex problems using super powerful computers called Deep Neural Networks (DNNs). These DNNs have billions of tiny parts, but most of these parts are not actually helping. To make them more efficient and faster, researchers want to get rid of the unnecessary parts. They’re using special techniques called attribution methods from a field called eXplainable AI (XAI) to figure out which parts are least important. By optimizing these methods, they can remove more parts and still keep the DNNs working well on big tasks like recognizing objects in pictures. Interestingly, they found that transformers (a type of DNN) have many more unnecessary parts than other types of DNNs. You can see their code and learn more at https://github.com/erfanhatefi/Pruning-by-eXplaining-in-PyTorch. |
Keywords
» Artificial intelligence » Classification » Few shot » Model compression » Pruning » Resnet » Transformer » Vit
