Summary of Online Policy Learning From Offline Preferences, by Guoxi Zhang and Han Bao and Hisashi Kashima
Online Policy Learning from Offline Preferences
by Guoxi Zhang, Han Bao, Hisashi Kashima
First submitted to arxiv on: 15 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed framework in this paper addresses generalizability issues in preference-based reinforcement learning (PbRL) by introducing virtual preferences that track an agent’s behaviors and provide well-aligned guidance. The approach combines offline preferences with virtual preferences, which are comparisons between the agent’s actions and offline data. This allows the learned reward function to be fitted on both offline and online data, enabling effective generalization in PbRL. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study improves preference-based reinforcement learning by allowing agents to learn from their own behaviors. The framework combines offline preferences with virtual preferences that track an agent’s actions, providing a better fit for real-world situations where agents may not follow offline data. By doing so, this approach enhances the generalizability of learned reward functions in PbRL. |
Keywords
* Artificial intelligence * Generalization * Reinforcement learning
