Summary of Tofueval: Evaluating Hallucinations Of Llms on Topic-focused Dialogue Summarization, by Liyan Tang et al.
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization
by Liyan Tang, Igor Shalyminov, Amy Wing-mei Wong, Jon Burnsky, Jake W. Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, Kathleen McKeown
First submitted to arxiv on: 20 Feb 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A recent surge in progress on faithful summarization has led to substantial advancements in evaluating factual consistency, also known as hallucinations. The paper investigates whether these breakthroughs transfer to other text summarization domains, specifically topic-focused dialogue summarization generated by Large Language Models (LLMs) of varying sizes. The authors introduce a new evaluation benchmark and provide binary sentence-level human annotations for the factual consistency of these summaries, along with detailed explanations of factually inconsistent sentences. Analysis reveals that existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of model size. Interestingly, when LLMs serve as binary factual evaluators, they perform poorly and can be outperformed by prevailing state-of-the-art specialized factuality evaluation metrics. The study also explores diverse error types with a curated error taxonomy, finding that there are various error distributions in model-generated summaries and that non-LLM based metrics can capture all error types better than LLM-based evaluators. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A new paper looks at how well large language models (LLMs) do at summarizing conversations. They compared these models to see if they’re good at getting the facts right. It turns out that even the biggest and best LLMs make mistakes and get things wrong. This is important because it means we need better ways to check if what a model says is true or not. The study also found that there are many different kinds of mistakes, and some methods are better at catching them than others. |
Keywords
» Artificial intelligence » Summarization
