Summary of Qapyramid: Fine-grained Evaluation Of Content Selection For Text Summarization, by Shiyue Zhang et al.
QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarization
by Shiyue Zhang, David Wan, Arie Cattan, Ayal Klein, Ido Dagan, Mohit Bansal
First submitted to arxiv on: 10 Dec 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper addresses a long-standing challenge in text summarization by proposing a new approach to evaluating the content selection of summarization models. The existing Pyramid protocol is widely used but lacks systematicity in defining and granulating sub-units. QAPyramid, the proposed method, breaks down reference summaries into finer-grained question-answer (QA) pairs based on the QA-SRL framework. This allows for more precise evaluation of content selection and maintains high inter-annotator agreement without requiring expert annotations. The authors collect QA-SRL annotations from CNN/DM and evaluate 10 summarization systems, resulting in a large dataset of over 8,900 QA-level annotations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about finding a better way to test how good text summaries are at choosing what’s important to include. Right now, there’s a widely used method called Pyramid that doesn’t work very well because it’s hard to define and measure what makes a good summary. The authors of this paper came up with a new idea called QAPyramid that breaks down the reference summaries into smaller pieces based on questions and answers. This helps make sure the evaluation is more accurate and fair, even if different people are doing the evaluating. |
Keywords
» Artificial intelligence » Cnn » Summarization
