Summary of Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-language Model Via Causality Analysis, by Po-hsuan Huang et al.
Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis
by Po-Hsuan Huang, Jeng-Lin Li, Chin-Po Chen, Ming-Ching Chang, Wei-Chao Chen
First submitted to arxiv on: 4 Dec 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates large vision-language models’ (LVLMs) tendency to generate non-existent visual elements, known as multimodal hallucination, which erodes user trust in their real-world applications. The authors hypothesize that hidden factors such as objects, contexts, and semantic foreground-background structures induce this hallucination. To address this issue, they propose a novel causal approach: a hallucination probing system to identify these hidden factors. By analyzing the causality between images, text prompts, and network saliency, they explore interventions to block these factors. Experimental results show that a straightforward technique based on their analysis can significantly reduce hallucinations, with potential for editing network internals to minimize hallucinated outputs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at big language models that can understand pictures and words. Sometimes these models make up fake things that aren’t really there, which is bad because people don’t trust them anymore. The researchers think it’s because of hidden things like objects or backgrounds that they’re not supposed to see. They came up with a new way to figure out what’s causing this problem and test some solutions to stop it from happening. They found that by using their method, they can make the models be less likely to make up fake things. |
Keywords
» Artificial intelligence » Hallucination
