Summary of The Dark Side Of Rich Rewards: Understanding and Mitigating Noise in Vlm Rewards, by Sukai Huang et al.
The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewards
by Sukai Huang, Shu-Wei Liu, Nir Lipovetzky, Trevor Cohn
First submitted to arxiv on: 24 Sep 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Robotics (cs.RO)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper challenges the conventional wisdom that Vision-Language Models (VLMs) are effective in guiding embodied agents to follow instructions. Instead, it reveals that agents trained using VLM rewards often underperform compared to those relying solely on intrinsic rewards. The authors hypothesize that false positive rewards, where unintended trajectories receive incorrect rewards, are more detrimental than false negatives. Analysis confirms this hypothesis, highlighting the widespread use of cosine similarity metric as prone to false positive reward estimates. To address this issue, the paper introduces BiMI (Binary Mutual Information), a novel reward function designed to mitigate noise. BiMI significantly enhances learning efficiency across diverse and challenging embodied navigation environments. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study looks at how we can train robots to follow instructions using computer vision and language. The researchers found that when we use special rewards to guide the robots, they don’t always do better than if we just let them explore on their own. In fact, sometimes they even do worse. This is because the rewards we give them can be wrong or misleading. To fix this problem, the authors came up with a new way to calculate these rewards that works better and helps the robots learn faster. |
Keywords
» Artificial intelligence » Cosine similarity
