Summary of Cross-modal Projection in Multimodal Llms Doesn’t Really Project Visual Attributes to Textual Space, by Gaurav Verma et al.
Cross-Modal Projection in Multimodal LLMs Doesn’t Really Project Visual Attributes to Textual Space
by Gaurav Verma, Minje Choi, Kartik Sharma, Jamelle Watson-Daniels, Sejoon Oh, Srijan Kumar
First submitted to arxiv on: 26 Feb 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the capabilities of multimodal large language models (MLLMs) in modeling domain-specific visual attributes. Specifically, it investigates how these models fine-tuned for specific domains like dermatology and agriculture perform on tasks related to images from those fields. The study uses four datasets and two fine-tuning settings to analyze the role of MLLMs’ two major modules: an image-language projection network and a large language model (LLM). The results show that as MLLMs are fine-tuned, they gain domain-specific visual capabilities, but the updates do not lead to the projection extracting relevant domain-specific visual attributes. Instead, the LLM is found to be modeling these attributes even when only the projection is fine-tuned. This study provides a potential reinterpretation of the role of cross-modal projections in MLLM architectures. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how big AI models can understand images about specific things like skin problems or farm equipment. These models are good at talking to humans, but they need to be trained for each special area to get really good. The researchers tested these models on different sets of pictures and found that when they’re trained, the models get better at understanding what’s in the pictures. But they don’t do it by looking at the images themselves; instead, they learn to understand the important details just by talking about the images. |
Keywords
» Artificial intelligence » Fine tuning » Large language model
