Summary of Meddoc-bot: a Chat Tool For Comparative Analysis Of Large Language Models in the Context Of the Pediatric Hypertension Guideline, by Mohamed Yaseen Jabarulla et al.
MedDoc-Bot: A Chat Tool for Comparative Analysis of Large Language Models in the Context of the Pediatric Hypertension Guideline
by Mohamed Yaseen Jabarulla, Steffen Oeltze-Jafra, Philipp Beerbaum, Theodor Uden
First submitted to arxiv on: 6 May 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A machine learning study evaluates the performance of four open-source large language models (LLMs) – Meditron, MedAlpaca, Mistral, and Llama-2 – in interpreting medical guidelines saved in PDF format. The researchers focus on hypertension guidelines for children and adolescents from the European Society of Cardiology (ESC). They develop a user-friendly chatbot tool using Python’s Streamlit library, allowing authorized users to upload PDF files and pose questions, generating responses from the four LLMs. A pediatric expert provides benchmarks by formulating questions and responses extracted from ESC guidelines, rating model-generated responses based on fidelity and relevance. The study assesses METEOR and chrF metric scores to evaluate similarity to reference answers. Results show that Llama-2 and Mistral performed well in metrics evaluation, while Llama-2 was slower when dealing with text and tabular data. In a human evaluation, responses from Mistral, Meditron, and Llama-2 exhibited reasonable fidelity and relevance. This study provides valuable insights into the strengths and limitations of LLMs for future developments in medical document interpretation. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how well four open-source language models work when trying to understand medical guidelines saved as PDF files. They focus on a specific set of guidelines about high blood pressure in children. The researchers make a special tool that allows authorized people to upload the guidelines and ask questions, getting answers from the four language models. A pediatric expert helps by coming up with questions and answers based on the guidelines, then rating how well the machine-generated answers match what they should be. The results show that two of the models did really well, but one was slower when working with text and tables. When people evaluated the answers, three of the models did a pretty good job of providing accurate and relevant information. This study helps us understand what these language models can do and how we can improve them. |
Keywords
» Artificial intelligence » Llama » Machine learning
