Summary of Cleanerclip: Fine-grained Counterfactual Semantic Augmentation For Backdoor Defense in Contrastive Learning, by Yuan Xun et al.
CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning
by Yuan Xun, Siyuan Liang, Xiaojun Jia, Xinwei Liu, Xiaochun Cao
First submitted to arxiv on: 26 Sep 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The abstract discusses the susceptibility of pre-trained large models for multimodal contrastive learning, such as CLIP, to data-poisoned backdoor attacks. It highlights that finetuning is a simpler and more efficient defense choice compared to retraining large models with augmented data. However, it also notes that existing fine-tuning defense strategies have limitations when facing complex attack techniques. To address this weakness, the authors propose TA-Cleaner, a fine-grained text alignment cleaner that cuts off feature connections of backdoor triggers by aligning subtexts to images and strengthening text self-supervision. The paper evaluates the effectiveness of TA-Cleaner against six attack algorithms and conducts comprehensive zero-shot classification tests on ImageNet1K, achieving state-of-the-art defensiveness among finetuning-based defense techniques. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The abstract is about making big models safer from bad data attacks. It says that these attacks are a problem because they can ruin the results of models like CLIP. The authors suggest using finetuning as a way to make models more robust, but this has its own limitations. To solve this issue, they propose TA-Cleaner, which helps prevent backdoor attacks by aligning text and images in a special way. They test it against different types of attacks and show that it works better than other methods. |
Keywords
» Artificial intelligence » Alignment » Classification » Fine tuning » Zero shot
