Summary of Mediq: Question-asking Llms and a Benchmark For Reliable Interactive Clinical Reasoning, by Shuyue Stella Li et al.
MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning
by Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, Yulia Tsvetkov
First submitted to arxiv on: 3 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a shift from traditional static benchmarks for language models to an interactive paradigm, where models proactively ask questions to gather more information and respond reliably. The authors develop the MediQ benchmark, simulating clinical interactions between a patient system and an adaptive expert system that elicits missing details via follow-up questions. They provide a pipeline to convert single-turn medical benchmarks into an interactive format and demonstrate that directly prompting state-of-the-art language models to ask questions degrades performance. To improve diagnostic accuracy, the authors experiment with abstention strategies and filtering irrelevant contexts, achieving 22.3% improvement. However, performance still lags compared to an unrealistic upper bound with complete information upfront. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper changes how we test language models by making them ask questions like doctors do in real-life conversations. Right now, most tests just give the model one question at a time and see if it gets the answer right. But that’s not how doctors work – they ask follow-up questions to get more information! The researchers created a new way of testing called MediQ, which simulates conversations between patients and doctors. They found that when language models are forced to ask questions, their performance drops. To fix this, they came up with ways for the models to decide when to ask questions and improved diagnostic accuracy by 22.3%. The goal is to make language models better at finding the information they need to give good answers. |
Keywords
» Artificial intelligence » Prompting
