Summary of Bridge the Modality and Capability Gaps in Vision-language Model Selection, by Chao Yi et al.
Bridge the Modality and Capability Gaps in Vision-Language Model Selection
by Chao Yi, Yu-Hang He, De-Chuan Zhan, Han-Jia Ye
First submitted to arxiv on: 20 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a method called VLM Selection With gAp Bridging (SWAB) to select the most suitable Pre-Trained Vision Language Models (VLMs) for zero-shot image classification tasks. The approach addresses two challenges: the “Modality Gap” between text and image embeddings, and the “Capability Gap” between general performance and dataset-specific performance. SWAB uses optimal transport to capture relevance between open-source datasets and target datasets, then transfers useful statistics from VLMs to bridge these gaps. This allows for accurate prediction of performance rankings without requiring access to test images. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps us pick the best computer models to classify pictures without showing them. We have a problem when we use text descriptions instead of pictures because our brains work differently with words and images. The new method, SWAB, makes it better by matching how well different models do on one task to how they’ll do on another. It’s like finding the right person for a job based on their past experiences. This helps us choose the best model for a specific task without needing all the pictures. |
Keywords
* Artificial intelligence * Image classification * Zero shot
