Summary of Towards Adversarially Robust Vision-language Models: Insights From Design Choices and Prompt Formatting Techniques, by Rishika Bhagwatkar et al.
Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
by Rishika Bhagwatkar, Shravan Nayak, Reza Bayat, Alexis Roger, Daniel Z Kaplan, Pouya Bashivan, Irina Rish
First submitted to arxiv on: 15 Jul 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the impact of Vision-Language Model (VLM) design choices on their robustness to image-based attacks, a critical concern as VLMs become increasingly prevalent. The authors introduce novel approaches to enhance robustness through prompt formatting, demonstrating significant improvements against strong attacks like Auto-PGD. Their findings provide important guidelines for developing more robust VLMs, essential for deployment in safety-critical environments. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how to make computer models that combine images and words (VLMs) safer from bad attacks. Right now, these models are really popular and used in many real-life situations. But it’s important to make sure they’re protected from fake or misleading information. The researchers did some experiments to see how different choices made when building the model affect its ability to resist attacks. They also came up with new ways to make the models more robust by tweaking the way questions are asked and potential fake images are created. Overall, their results show that making these changes can help protect VLMs from strong attacks. |
Keywords
* Artificial intelligence * Language model * Prompt
