Summary of Code As Reward: Empowering Reinforcement Learning with Vlms, by David Venuto et al.
Code as Reward: Empowering Reinforcement Learning with VLMs
by David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, Ankit Anand
First submitted to arxiv on: 7 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces Code as Reward (VLM-CaR), a framework that leverages pre-trained Vision-Language Models (VLMs) to support the training of reinforcement learning (RL) agents. VLMs can analyze visual observations and provide feedback on task completion, making them suitable for RL tasks. However, querying VLMs directly is computationally expensive, so VLM-CaR generates dense reward functions from VLMs through code generation, reducing the computational burden. The paper shows that these generated rewards are accurate across diverse environments and can be more effective in training RL policies than original sparse rewards. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you’re teaching a robot to do new tasks, like picking up objects or navigating mazes. This paper explores how to use special computer models called Vision-Language Models (VLMs) to help the robot learn faster and better. VLMs are good at understanding pictures and giving feedback on what’s happening. But it takes a lot of computer power to ask them for help all the time, so this paper creates a new way to get rewards from the VLM without slowing down the robot. The results show that this approach can be very helpful in teaching robots to do new things. |
Keywords
* Artificial intelligence * Reinforcement learning
