Summary of German Also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset, by Laura Mascarell et al.
German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset
by Laura Mascarell, Ribin Chalumattu, Annette Rios
First submitted to arxiv on: 6 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The advent of Large Language Models (LLMs) has led to significant progress in natural language processing tasks, but they still struggle with hallucinating information in their output. This issue is particularly critical for automatic text summarization, where consistency between the generated summary and source document is crucial. To address this challenge, previous research focused on detecting hallucinations (inconsistency detection) to evaluate the faithfulness of generated summaries. However, these works primarily targeted English language, with recent multilingual approaches lacking German data. This paper presents absinth, a manually annotated dataset for hallucination detection in German news summarization, and explores the capabilities of novel open-source LLMs on this task through fine-tuning and in-context learning settings. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large Language Models (LLMs) have made great progress in natural language processing tasks. But they still make mistakes by adding information that isn’t really there. This is a big problem for automatic text summarization, where we want the generated summary to match the original document. To fix this issue, previous research tried to detect when these models make mistakes (inconsistency detection) so we can evaluate how accurate their summaries are. However, most of these efforts focused on English language and didn’t include German data. This paper solves this problem by creating a dataset called absinth for detecting hallucinations in German news summarization and tests open-source LLMs to see what they can do. |
Keywords
» Artificial intelligence » Fine tuning » Hallucination » Natural language processing » Summarization
