Summary of Fine-tuning Clip Text Encoders with Two-step Paraphrasing, by Hyunjae Kim et al.
Fine-tuning CLIP Text Encoders with Two-step Paraphrasing
by Hyunjae Kim, Seunghyun Yoon, Trung Bui, Handong Zhao, Quan Tran, Franck Dernoncourt, Jaewoo Kang
First submitted to arxiv on: 23 Feb 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a straightforward approach to enhance the representations of Contrastive Language-Image Pre-training (CLIP) models for paraphrases. The CLIP model is initially trained on natural language input to produce accurate visual outputs, but it struggles with linguistic variations in input queries, such as paraphrases. To address this limitation, the authors introduce a two-step paraphrase generation process using large language models and fine-tune the CLIP text encoder using these generated paraphrases while freezing the image encoder. The resulting model, ParaCLIP, shows significant improvements over baseline CLIP models across various tasks, including paraphrased retrieval, Visual Genome Relation and Attribution, and seven semantic textual similarity tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps computers better understand language by making a special kind of AI model called Contrastive Language-Image Pre-training (CLIP) work better with different phrases that mean the same thing. Currently, CLIP models are good at understanding simple language, but they get confused when people use different words to say the same thing. To fix this, the researchers created a way to make CLIP models practice understanding these different phrases by generating lots of examples and then training the model on them. This new model, called ParaCLIP, is much better than the original at tasks like finding pictures that match a description. |
Keywords
* Artificial intelligence * Encoder
