Summary of Connecting the Dots: Collaborative Fine-tuning For Black-box Vision-language Models, by Zhengbo Wang et al.
Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models
by Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, Tieniu Tan
First submitted to arxiv on: 6 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach for fine-tuning black-box vision-language models (VLMs) is proposed in this paper, addressing the challenge of model owners safeguarding their intellectual property by providing only input prompts and output predictions. The Collaborative Fine-Tuning (CraFT) method consists of two modules: prompt generation and prediction refinement, optimized using a collaborative training algorithm. Experimental results on 15 datasets demonstrate CraFT’s superiority, achieving a 12% gain with 16-shot datasets while requiring significantly less memory and computational resources compared to white-box methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper explores ways to fine-tune vision-language models without accessing their internal workings. The authors propose an innovative approach called Collaborative Fine-Tuning (CraFT), which generates prompts and refines predictions. This method is tested on many datasets, showing that it can do a better job than other methods while using less resources. |
Keywords
* Artificial intelligence * Fine tuning * Prompt
