Summary of Prompt-based Visual Alignment For Zero-shot Policy Transfer, by Haihan Gao et al.
Prompt-based Visual Alignment for Zero-shot Policy Transfer
by Haihan Gao, Rui Zhang, Qi Yi, Hantao Yao, Haochen Li, Jiaming Guo, Shaohui Peng, Yunkai Gao, QiCheng Wang, Xing Hu, Yuanbo Wen, Zihao Zhang, Zidong Du, Ling Li, Qi Guo, Yunji Chen
First submitted to arxiv on: 5 Jun 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes Prompt-Based Visual Alignment (PVA), a framework for mitigating domain bias in reinforcement learning (RL) and achieving zero-shot policy transfer across multiple domains. By leveraging the semantic information contained in text sequences, PVA trains a visual aligner that maps images from diverse domains to a unified representation, enabling good generalization performance. The approach relies on prompt tuning to learn a sequence of learnable tokens that better depict semantic information. This framework addresses existing limitations by providing explicit semantic constraints for the feature extractor and achieving robust zero-shot generalization in unseen domains. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper presents a solution to overcome the issue of overfitting in RL, which hinders applications in reinforcement learning. The proposed method, Prompt-Based Visual Alignment (PVA), helps the agent learn a unified cross-domain representation by using semantic information from text sequences as constraints. This approach allows for good generalization performance and achieves zero-shot policy transfer across multiple domains. |
Keywords
» Artificial intelligence » Alignment » Generalization » Overfitting » Prompt » Reinforcement learning » Zero shot
