Summary of Few-shot Adversarial Prompt Learning on Vision-language Models, by Yiwei Zhou et al.
Few-Shot Adversarial Prompt Learning on Vision-Language Models
by Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, Tongliang Liu
First submitted to arxiv on: 21 Mar 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Computation and Language (cs.CL); Cryptography and Security (cs.CR); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper tackles the issue of deep neural networks being vulnerable to imperceptible adversarial perturbations. Building on the success of vision-language foundation models, previous attempts achieved zero-shot adversarial robustness by aligning adversarial visual features with text supervision. However, these efforts were limited due to high adaptation costs, suboptimal text supervision, and uncontrolled natural generalization capacity. To address these issues, this paper proposes a few-shot adversarial prompt framework that adapts input sequences using limited data to achieve significant robustness improvements. The approach involves providing end-to-end learned adversarially correlated text supervision from adversarial examples. A novel training objective is also introduced to enhance consistency in multi-modal features and encourage differentiated uni-modal features between natural and adversarial examples. The proposed framework enables learning adversarial text supervision, achieving superior cross-modal adversarial alignment and matching state-of-the-art zero-shot robustness with just 1% training data. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research paper is about making deep neural networks more secure against tiny attacks that are hard to detect. Current methods can already do this, but they have some limitations, like needing a lot of extra data and not being very good at generalizing. To solve these problems, the researchers propose a new way to make these networks more robust using only a little bit of training data. The key idea is to use text information that is specifically designed to work with adversarial attacks. They also introduce a new training method that helps the network learn better features and separate natural and attacked images. Overall, this research provides a more effective and efficient way to protect neural networks from small but sneaky attacks. |
Keywords
* Artificial intelligence * Alignment * Few shot * Generalization * Multi modal * Prompt * Zero shot
