Summary of Sample-specific Masks For Visual Reprogramming-based Prompting, by Chengyi Cai and Zesheng Ye and Lei Feng and Jianzhong Qi and Feng Liu
Sample-specific Masks for Visual Reprogramming-based Prompting
by Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, Feng Liu
First submitted to arxiv on: 5 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a novel approach to visual reprogramming (VR) that addresses the limitations of existing methods. VR is a technique used to repurpose pre-trained models for new tasks by adding small-scale patterns into input images. The authors show that the shared mask used in current VR methods can limit generalization and increase approximation error due to a lack of sample-level adaptation. To overcome this limitation, they propose a new framework called Sample-Specific Multi-Channel Masks (SMM), which generates individual masks for each sample using a lightweight ConvNet and patch-wise interpolation. SMM is theoretically shown to reduce approximation error compared to existing state-of-the-art VR methods. The authors empirically demonstrate the performance gain of SMM on both ResNet and ViT. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper makes a new technique called visual reprogramming better. Visual reprogramming takes pre-trained models and adds small patterns to make them work for different tasks. But, this method has some problems. It uses the same pattern for every picture, which limits how well it works. To fix this, the authors created a new way to add patterns to pictures that is better than before. This new way is called Sample-Specific Multi-Channel Masks (SMM). SMM makes different patterns for each picture, which makes it work better. The authors tested SMM and showed that it does a better job than other methods. |
Keywords
» Artificial intelligence » Generalization » Mask » Resnet » Vit
