Summary of Llms Hallucinate Graphs Too: a Structural Perspective, by Erwan Le Merrer and Gilles Tredan
LLMs hallucinate graphs too: a structural perspective
by Erwan Le Merrer, Gilles Tredan
First submitted to arxiv on: 30 Aug 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Social and Information Networks (cs.SI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed study investigates the phenomenon of hallucinations in Large Language Models (LLMs) by analyzing incorrect outputs when prompted with well-known graphs from literature. The research introduces a novel approach to characterize these hallucinations using graph theory, demonstrating the potential for rich hallucinated graphs to provide insights into LLM outputs. This is achieved through two primary contributions: first, observing the diversity of topological hallucinations from major modern LLMs; and second, proposing a metric for measuring the amplitude of such hallucinations – the Graph Atlas Distance. The study also compares this metric with the Hallucination Leaderboard, which leverages 10,000 times more prompts to obtain its ranking. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research looks into how language models make mistakes when showing facts about graphs from famous books and movies like Les Misérables or the Karate club. It finds that these models often create new incorrect graphs instead of just saying “I don’t know”. This study shows that by looking at these incorrect graphs, we can learn more about what’s going on inside language models’ minds. Two main ideas are explored: first, how different models make mistakes in this way; and second, a special measure to see how big these mistakes are – like measuring the distance between real and fake graphs. |
Keywords
* Artificial intelligence * Hallucination
