Summary of Preference-guided Reinforcement Learning For Efficient Exploration, by Guojian Wang et al.
Preference-Guided Reinforcement Learning for Efficient Exploration
by Guojian Wang, Faguo Wu, Xiao Zhang, Tianyuan Chen, Xuyang Chen, Lin Zhao
First submitted to arxiv on: 9 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates preference-based reinforcement learning (PbRL) for agents to learn from human feedback. The approach is valuable when defining a fine-grain reward function is not feasible, but it’s inefficient and impractical for promoting deep exploration in hard-exploration tasks with long horizons and sparse rewards. To tackle this issue, the authors introduce LOPE: Learning Online with trajectory Preference guidancE, an end-to-end preference-guided RL framework that enhances exploration efficiency. LOPE includes a two-step sequential policy optimization process consisting of trust-region-based policy improvement and preference guidance steps. The authors also reformulate preference guidance as a novel trajectory-wise state marginal matching problem that minimizes the maximum mean discrepancy distance between preferred trajectories and the learned policy. A theoretical analysis is provided to characterize the performance improvement bound, and the effectiveness of LOPE is evaluated in various challenging hard-exploration environments. LOPE outperforms several state-of-the-art methods regarding convergence rate and overall performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary PbRL allows agents to learn from human feedback, but it’s not great for exploring new areas. The authors create a new way called LOPE that makes exploration more efficient. They use two steps: one to improve the policy and another to adjust it based on what humans like. They also make sure their approach is fair by comparing the preferred trajectories with what the agent has learned. This new method does better than other methods in hard situations. |
Keywords
* Artificial intelligence * Optimization * Reinforcement learning
