Summary of Modeling Caption Diversity in Contrastive Vision-language Pretraining, by Samuel Lavoie et al.
Modeling Caption Diversity in Contrastive Vision-Language Pretraining
by Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mahmoud Assran, Andrew Gordon Wilson, Aaron Courville, Nicolas Ballas
First submitted to arxiv on: 30 Apr 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Contrastive Language Pretraining (CLIP) models an image and its caption as a single vector, limiting representation diversity. This paper introduces Llip, Latent Language Image Pretraining, which generates diverse captions for an image by conditioning on text-derived information. Llip’s vision encoder outputs visual features mixed with textual information to produce a final representation. The proposed method outperforms non-contextualized baselines like CLIP and SigLIP on various tasks, even with large-scale encoders. Notably, Llip improves zero-shot classification by 2.9% and achieves a top-1 accuracy of 83.5% on ImageNet, surpassing a similarly sized CLIP by 1.4%. The method also demonstrates improved performance on zero-shot retrieval on MS-COCO by 6.0%. A comprehensive analysis of Llip’s components shows that it leads to richer visual representations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you want to describe an image in many different ways. This paper proposes a new way to do this, called Llip. Unlike other methods that try to match images with captions by using a single vector, Llip tries to capture the many different ways people could describe an image. The authors show that Llip is better than other methods at tasks like classifying images and finding similar images. They also show that Llip can improve how well computers understand what’s in an image. |
Keywords
» Artificial intelligence » Classification » Encoder » Pretraining » Zero shot
