Summary of C-tpt: Calibrated Test-time Prompt Tuning For Vision-language Models Via Text Feature Dispersion, by Hee Suk Yoon et al.
C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion
by Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark Hasegawa-Johnson, Yingzhen Li, Chang D. Yoo
First submitted to arxiv on: 21 Mar 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper explores the idea of calibration in deep learning, specifically for large-scale vision-language models like CLIP. The authors propose a method called Calibrated Test-time Prompt Tuning (C-TPT) that can optimize prompts during test-time without requiring labeled data. The key insight is that the prompt choice affects the calibration of predictions, with prompts leading to higher text feature dispersion resulting in better-calibrated predictions. The authors introduce the Average Text Feature Dispersion (ATFD) metric and show its relationship with calibration error. They demonstrate the effectiveness of C-TPT on different CLIP architectures and datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about how to make sure that AI models are accurate, not just good at guessing. It’s like taking a test – you want to know if your answers are correct or not. The authors found that when they use special words (called prompts) to help the model understand what it’s seeing, it gets better at predicting things correctly. But they also noticed that these prompts can make the model less sure about its predictions, which is important to know. They came up with a new way to fine-tune the prompts so that the model is more accurate and confident in its answers. This method doesn’t need any special training data, just the words it’s already learned. |
Keywords
* Artificial intelligence * Deep learning * Prompt
