Summary of Tima: Text-image Mutual Awareness For Balancing Zero-shot Adversarial Robustness and Generalization Ability, by Fengji Ma et al.
TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability
by Fengji Ma, Li Liu, Hei Victor Cheng
First submitted to arxiv on: 27 May 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach is proposed to achieve zero-shot adversarial robustness in large-scale foundation models like Contrastive Language-Image Pre-training (CLIP), which excel at zero-shot generalization but are vulnerable to adversarial perturbations. Existing methods balance robustness and generalization under small perturbations, but fail to do so under larger ones. The proposed Text-Image Mutual Awareness (TIMA) method strikes a balance by incorporating Minimum Hyperspherical Energy (MHE) into text embeddings and using fixed pre-trained image embeddings as cross-modal auxiliary supervision for knowledge distillation. Additionally, the TAI tuning mechanism increases inter-class distance between image embeddings during training using Text-distance based Adaptive Margin (TAM). Experimental results demonstrate the effectiveness of TIMA, achieving impressive zero-shot performance against various adversarial perturbations while preserving generalization capabilities. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large-scale foundation models like CLIP are great at recognizing things without being taught, but they’re very bad at handling fake or misleading information. To fix this, researchers propose a new way to make these models more robust against bad data. They use two techniques: one for text and one for images. The text technique makes words that aren’t related close together, making it harder for fake information to slip through. The image technique does the same thing, but for pictures. By combining these two techniques, they show that their method is really good at keeping the model safe from bad data. |
Keywords
* Artificial intelligence * Generalization * Knowledge distillation * Zero shot
