Summary of It’s Not a Modality Gap: Characterizing and Addressing the Contrastive Gap, by Abrar Fahim et al.
It’s Not a Modality Gap: Characterizing and Addressing the Contrastive Gap
by Abrar Fahim, Alex Murphy, Alona Fyshe
First submitted to arxiv on: 28 May 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Computation and Language (cs.CL); Information Retrieval (cs.IR); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the limitations of multi-modal contrastive models like CLIP, which excel in zero-shot classification by projecting images and texts onto a shared latent space. However, a recent discovery indicates that two-encoder contrastive models like CLIP exhibit a modality gap, where image and text embeddings reside in distinct areas of the latent space. The study reveals that this gap persists even when accounting for factors like the cone effect, mismatched pairs, and insufficient training. Instead, the authors propose that the modality gap is an inherent property of the two-encoder contrastive loss, which they rename as the contrastive gap. By analyzing the uniformity and alignment properties of CLIP’s latent space, the researchers attribute the contrastive gap to low uniformity, resulting in embeddings occupying a small portion of the space. To address this issue, the authors modify the contrastive loss to distribute embeddings more uniformly, achieving better performance in downstream tasks like zero-shot image classification and multi-modal arithmetic. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study examines how CLIP and similar models can struggle with combining images and text. Even when these models do well at first glance, they may not be using their representations efficiently. The team behind this research found that the problem is due to how the model’s “space” is arranged, making it harder for different types of information (like images and text) to work together effectively. They suggest a new way to adjust the model so that these different types of information can coexist better. |
Keywords
» Artificial intelligence » Alignment » Classification » Contrastive loss » Encoder » Image classification » Latent space » Multi modal » Zero shot
