Summary of Understanding Figurative Meaning Through Explainable Visual Entailment, by Arkadiy Saakyan et al.
Understanding Figurative Meaning through Explainable Visual Entailment
by Arkadiy Saakyan, Shreyas Kulkarni, Tuhin Chakrabarty, Smaranda Muresan
First submitted to arxiv on: 2 May 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper investigates the capabilities of Large Vision-Language Models (VLMs) in understanding figurative meaning in images and captions. Specifically, it introduces a new task called explainable visual entailment, where VLMs must predict whether an image entails a caption and justify their prediction with a textual explanation. The paper presents a dataset, V-FLUTE, containing 6,027 instances of images, captions, labels, and explanations spanning five figurative phenomena: metaphors, similes, idioms, sarcasm, and humor. The results show that VLMs struggle to generalize from literal to figurative meaning, particularly when the figurative meaning is present in the image. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large Vision-Language Models are super smart computers that can understand what’s happening in pictures and words. But they’re not great at understanding jokes or figures of speech like metaphors or sarcasm. This paper tries to fix that by creating a new task where these models have to figure out if an image makes sense with a caption, and explain why it does or doesn’t make sense. The researchers made a special dataset with lots of examples of images, captions, and explanations for five types of figurative language: metaphors, similes, idioms, sarcasm, and humor. They found that these models are really bad at understanding figurative language, especially when it’s in the pictures. |
