Summary of Ravl: Discovering and Mitigating Spurious Correlations in Fine-tuned Vision-language Models, by Maya Varma et al.
RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models
by Maya Varma, Jean-Benoit Delbrouck, Zhihong Chen, Akshay Chaudhari, Curtis Langlotz
First submitted to arxiv on: 6 Nov 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a novel approach called RaVL to address spurious correlations in fine-tuned vision-language models (VLMs). Existing methods operate at the global image level, neglecting fine-grained features that contribute to zero-shot performance degradation. RaVL discovers and mitigates spurious correlations by leveraging region-level clustering to identify precise image features causing errors. A novel region-aware loss function is then used to fine-tune the VLM, focusing on relevant regions and ignoring spurious relationships. The paper evaluates RaVL on 654 VLMs with various architectures, data domains, and learned spurious correlations, demonstrating a significant improvement in discovering (191%) and mitigating (8.2% improvement) spurious correlations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary RaVL is a new way to make computer vision models better at understanding pictures. Right now, these models can get confused by things they shouldn’t be looking at. This paper shows how RaVL finds and fixes these problems using special techniques that focus on specific parts of the picture rather than just the whole thing. They tested RaVL with many different types of images and models, and it worked really well, making the models better at recognizing pictures. |
Keywords
» Artificial intelligence » Clustering » Loss function » Zero shot
