Summary of Test-time Adaptation with Salip: a Cascade Of Sam and Clip For Zero Shot Medical Image Segmentation, by Sidra Aleem et al.
Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentation
by Sidra Aleem, Fangyijie Wang, Mayug Maniparambil, Eric Arazo, Julia Dietlmeier, Guenole Silvestre, Kathleen Curran, Noel E. O’Connor, Suzanne Little
First submitted to arxiv on: 9 Apr 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The Segment Anything Model (SAM) and CLIP, two remarkable vision foundation models, excel in segmentation tasks across diverse domains. This paper proposes a unified framework, SaLIP, that integrates SAM and CLIP for medical image segmentation without requiring extensive data or prior prompts. The framework involves using SAM for part-based segmentation within the image, followed by CLIP to retrieve the mask corresponding to the region of interest (ROI) from the pool of SAM-generated masks. Finally, SAM is prompted by the retrieved ROI to segment a specific organ. The method shows substantial enhancements in zero-shot segmentation, with notable improvements in DICE scores across diverse segmentation tasks like brain, lung, and fetal head. This framework is training- and fine-tuning-free, does not rely on domain expertise or labeled data for prompt engineering, and showcases the potential of SAM and CLIP in medical image segmentation. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary SAM and CLIP are two powerful vision foundation models that excel in different areas. This paper shows how to use them together to improve medical image segmentation without needing a lot of data or special training. The SaLIP framework works by using SAM to divide the image into parts, then using CLIP to find the right mask for the part you’re interested in. Finally, SAM is used again to segment a specific organ. This method works well and can even improve some segmentation tasks without needing any extra training or data. |
Keywords
» Artificial intelligence » Fine tuning » Image segmentation » Mask » Prompt » Sam » Zero shot
