Summary of Adapting Llms For the Medical Domain in Portuguese: a Study on Fine-tuning and Model Evaluation, by Pedro Henrique Paiola et al.
Adapting LLMs for the Medical Domain in Portuguese: A Study on Fine-Tuning and Model Evaluation
by Pedro Henrique Paiola, Gabriel Lino Garcia, João Renato Ribeiro Manesco, Mateus Roder, Douglas Rodrigues, João Paulo Papa
First submitted to arxiv on: 30 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The study evaluates the performance of large language models (LLMs) as medical agents in Portuguese, aiming to develop a reliable virtual assistant for healthcare professionals. The authors fine-tune LLMs using the PEFT-QLoRA method on datasets translated from English using GPT-3.5 and the MedQuAD dataset. They find that the InternLM2 model performs well, achieving high precision and adequacy in metrics such as accuracy, completeness, and safety. However, DrBode models exhibit catastrophic forgetting of acquired medical knowledge, although they perform well in grammaticality and coherence. The study highlights the need for robust assessment protocols due to low inter-rater agreement. The work paves the way for future research on multilingual models specific to the medical field, improving training data quality, and developing consistent evaluation methodologies. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models (LLMs) are being tested as virtual assistants in healthcare. Researchers took popular models like ChatBode-7B and InternLM2 and used them to help doctors with tasks. They gave the models information from medical databases and saw how well they did. Some models forgot what they learned, but others were great at grammar and making sense. The study shows that these models can be useful, but we need better ways to test them. |
Keywords
» Artificial intelligence » Gpt » Precision
