Summary of Mmedpo: Aligning Medical Vision-language Models with Clinical-aware Multimodal Preference Optimization, by Kangyu Zhu et al.
MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization
by Kangyu Zhu, Peng Xia, Yun Li, Hongtu Zhu, Sheng Wang, Huaxiu Yao
First submitted to arxiv on: 9 Dec 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The advancement of Large Vision-Language Models (LVLMs) has enabled their application in the medical field, but Medical LVLMs (Med-LVLMs) still face factuality challenges due to modality misalignment. Previous attempts to enhance modality alignment through preference optimization have inadequately mitigated clinical relevance, making samples easily distinguishable and reducing alignment effectiveness. To address this challenge, we propose MMedPO, a novel multimodal medical preference optimization approach that considers clinical relevance to enhance Med-LVLM alignment. MMedPO curates multimodal preference data by introducing dispreference types (plausible hallucinations and lesion region neglect) and calculates clinical relevance scores from multiple Med-LLMs and visual tools. We then integrate these scores into the preference optimization process as weights, enabling effective alignment. Our experiments demonstrate that MMedPO significantly enhances factual accuracy in Med-LVLMs, achieving substantial improvements over existing methods by averaging 14.2% and 51.7% across tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Medical Large Vision-Language Models are being used to help doctors with medical tasks, but they have a big problem: they’re not very accurate because they prioritize text information over visual information from medical images. Researchers tried to fix this by changing what the models “prefer” to learn, but that didn’t work well enough. To solve this issue, scientists came up with a new way to teach these models called MMedPO. It makes the models learn by showing them examples of when they got things wrong and correcting them. This approach worked much better than previous methods, improving the accuracy of medical predictions by 14-52%. |
Keywords
» Artificial intelligence » Alignment » Optimization
