Summary of Answering Real-world Clinical Questions Using Large Language Model Based Systems, by Yen Sia Low (1) et al.

Answering real-world clinical questions using large language model based systems

by Yen Sia Low, Michael L. Jackson, Rebecca J. Hyde, Robert E. Brown, Neil M. Sanghavi, Julian D. Baldwin, C. William Pike, Jananee Muralidharan, Gavin Hui, Natasha Alexander, Hadeel Hassan, Rahul V. Nene, Morgan Pike, Courtney J. Pokrzywa, Shivam Vedak, Adam Paul Yan, Dong-han Yao, Amy R. Zipursky, Christina Dinh, Philip Ballentine, Dan C. Derieg, Vladimir Polony, Rehan N. Chawdry, Jordan Davies, Brigham B. Hyde, Nigam H. Shah, Saurabh Gombar

First submitted to arxiv on: 29 Jun 2024

GrooveSquid.com Paper Summaries

GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!

Summary difficulty	Written by	Summary
High	Paper authors	High Difficulty Summary Read the original abstract here
Medium	GrooveSquid.com (original content)	Medium Difficulty Summary A new study evaluates the ability of large language models (LLMs) to answer clinical questions, with potential applications in healthcare decision-making. Researchers tested five LLM-based systems, including general-purpose models like ChatGPT-4 and Claude 3 Opus, as well as retrieval-augmented generation (RAG)-based and agentic models like OpenEvidence and ChatRWD. The results show that while general-purpose LLMs struggle to produce relevant and evidence-based answers (only 2% to 10%), RAG-based and agentic models perform better (24% to 58%). Notably, the agentic ChatRWD model is able to answer novel questions effectively (65% vs. 0-9%). These findings suggest that while general-purpose LLMs are not suitable for evidence summarization, purpose-built systems combining RAG and agency could improve access to pertinent evidence for patient care.
Low	GrooveSquid.com (original content)	Low Difficulty Summary Large language models (LLMs) might help doctors make better decisions by providing relevant medical information. Researchers tested five types of LLMs to see how well they answered clinical questions. Some LLMs were good at finding answers in existing research, while others could generate new studies based on real-world data. The results show that most LLMs didn’t do very well, but some did better than others. A special type of LLM called ChatRWD was able to answer questions it had never seen before! This could be helpful for doctors who need quick answers about patient care.