Summary of Muap: Multi-step Adaptive Prompt Learning For Vision-language Model with Missing Modality, by Ruiting Dai et al.
MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modality
by Ruiting Dai, Yuqiao Tan, Lisi Mo, Tao He, Ke Qin, Shuang Liang
First submitted to arxiv on: 7 Sep 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The recently proposed prompt learning approach has shown impressive results in various Vision-Language (VL) tasks, but existing models primarily focus on studying prompt generation and strategies with complete modality settings. This paper investigates how these models behave when modalities are incomplete, revealing their high sensitivity to missing modalities. To address this issue, the authors propose a novel Multi-step Adaptive Prompt Learning (MuAP) framework that generates multimodal prompts and performs multi-step prompt tuning. The MuAP framework adaptively learns knowledge by iteratively aligning modalities, mitigating the imbalance issue caused by only textual prompts. Extensive experiments demonstrate the effectiveness of MuAP, achieving significant improvements compared to state-of-the-art models on all benchmark datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper explores how machines learn from incomplete information in Vision-Language tasks. Existing methods are great at generating prompts and using them to solve problems, but they don’t work well when some information is missing. The authors develop a new way to adapt these methods to handle incomplete information. They test their approach on various datasets and show that it works better than existing methods. |
Keywords
» Artificial intelligence » Prompt
