Summary of Two-step Offline Preference-based Reinforcement Learning with Constrained Actions, by Yinglun Xu et al.
Two-Step Offline Preference-Based Reinforcement Learning with Constrained Actions
by Yinglun Xu, Tarun Suresh, Rohan Gumaste, David Zhu, Ruirui Li, Zhengyang Wang, Haoming Jiang, Xianfeng Tang, Qingyu Yin, Monica Xiao Cheng, Qi Zeng, Chao Zhang, Gagandeep Singh
First submitted to arxiv on: 30 Dec 2023
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel two-step learning framework, called PRC (Preference-Based Reinforcement Learning with Constrained Actions), is proposed to overcome challenges in preference-based reinforcement learning in offline settings. The framework limits the optimization space to a constrained action set that excludes out-of-distribution state-actions, addressing issues of reward hacking and complexity. Empirical results on robotic control environments demonstrate high learning efficiency across various datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A new way to teach machines how to make decisions is developed. This approach, called PRC, helps machines learn from experience without getting stuck in bad habits or trying to cheat. The method works by giving the machine a set of actions it can take and then adjusting its choices based on what it has learned so far. This makes the learning process more efficient and reliable. The method is tested on robots that control other devices, and the results show that PRC is effective in helping machines learn quickly and accurately. |
Keywords
* Artificial intelligence * Optimization * Reinforcement learning
