Summary of I Could’ve Asked That: Reformulating Unanswerable Questions, by Wenting Zhao et al.
I Could’ve Asked That: Reformulating Unanswerable Questions
by Wenting Zhao, Ge Gao, Claire Cardie, Alexander M. Rush
First submitted to arxiv on: 24 Jul 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper addresses a long-standing issue with large language models (LLMs) that can identify unanswerable questions in documents but fail to assist users in reformulating them. The authors introduce CouldAsk, an evaluation benchmark designed to study this problem. They evaluate state-of-the-art open-source and proprietary LLMs on CouldAsk, including GPT-4 and Llama2-7B, and find that these models are limited in their ability to reformulate questions correctly. Specifically, the results show that only 26% of GPT-4’s attempts and 12% of Llama2-7B’s attempts were successful. Error analysis reveals that most unsuccessful reformulations arise from the models simply rephrasing or generating identical questions. The authors publicly release the benchmark and code to reproduce the experiments, highlighting the need for more effective question reformulation capabilities in LLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you’re trying to find an answer to a question but can’t find it anywhere. Existing language models can tell when a question is unanswerable, but they don’t help you rephrase the question to make it easier to find the answer. This paper introduces a new way to test how well language models can reformulate questions that are hard or impossible to answer. The researchers tested popular language models and found that most of them struggle to come up with better questions. They think this is because these models just repeat what you asked or come up with identical questions. The authors hope their work will inspire new approaches to help people find the answers they need. |
Keywords
* Artificial intelligence * Gpt
