Summary of Leveraging Skills From Unlabeled Prior Data For Efficient Online Exploration, by Max Wilcoxson et al.
Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
by Max Wilcoxson, Qiyang Li, Kevin Frans, Sergey Levine
First submitted to arxiv on: 23 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel unsupervised pretraining approach for reinforcement learning (RL) is proposed, which leverages unlabeled offline trajectory data to learn efficient exploration strategies. The method, called SUPE (Skills from Unlabeled Prior data for Exploration), combines the benefits of low-level skills extraction using a variational autoencoder (VAE) and pseudo-labeling of unlabeled trajectories with optimistic rewards and high-level action labels. These transformed examples are then used as additional off-policy data for online RL to learn a high-level policy that composes pretrained low-level skills to explore efficiently. Experimental results demonstrate the effectiveness of SUPE, which consistently outperforms prior strategies across a suite of 42 long-horizon, sparse-reward tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A new way to help robots and computers learn from experience is developed in this paper. It’s called SUPE, and it uses old data that wasn’t labeled or supervised to teach machines how to explore and find good solutions. This approach combines two ideas: first, learning simple skills using an algorithm like a VAE, and then using those skills to understand what happened in the past and make better choices for the future. The result is a way to learn quickly and efficiently, even when there’s not much reward or feedback. In tests, SUPE worked well on many different tasks that required exploring and finding good solutions. |
Keywords
» Artificial intelligence » Pretraining » Reinforcement learning » Supervised » Unsupervised » Variational autoencoder
