Summary of Llava-surg: Towards Multimodal Surgical Assistant Via Structured Surgical Video Learning, by Jiajie Li et al.
LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning
by Jiajie Li, Garrett Skinner, Gene Yang, Brian R Quaranto, Steven D Schwaitzberg, Peter C W Kim, Jinjun Xiong
First submitted to arxiv on: 15 Aug 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces a novel approach to create a large-scale dataset for surgical video-instruction pairs, addressing the lack of datasets in this field. The authors propose a two-stage question-answer generation pipeline using large language models (LLMs) to learn surgical knowledge from publicly available videos. This pipeline reduces task complexity and mitigates LLM hallucinations. The generated dataset, Surg-QA, consists of 102,000 pairs, the largest of its kind. The authors also develop LLaVA-Surg, a vision-language conversational assistant trained on this dataset, which significantly outperforms general-domain models in zero-shot surgical video question-answering tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper creates a new dataset for surgical videos and trains an AI model to answer questions about them. This is important because doctors need help learning from videos of surgeries. The authors used a special type of AI called a large language model (LLM) to create the dataset and train the model. They made sure the LLM didn’t make up fake answers by breaking down the question-answering process into smaller steps. The resulting model, LLaVA-Surg, is really good at answering questions about surgical videos. |
Keywords
» Artificial intelligence » Large language model » Question answering » Zero shot
