Summary of Optimistic Model Rollouts For Pessimistic Offline Policy Optimization, by Yuanzhao Zhai et al.
Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
by Yuanzhao Zhai, Yiying Li, Zijian Gao, Xudong Gong, Kele Xu, Dawei Feng, Ding Bo, Huaimin Wang
First submitted to arxiv on: 11 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel model-based offline reinforcement learning (RL) framework called ORPO, which generates Optimistic model Rollouts for Pessimistic offline policy Optimization. The existing pessimistic approach, constructing a P-MDP, discourages policies from learning in out-of-distribution regions, under-utilizing the generalization ability of dynamics models. In contrast, ORPO constructs an O-MDP to encourage more optimistic rollouts and train an optimistic rollout policy. The relabeled state-action pairs are then used to optimize the output policy in the P-MDP. Experimental results show that ORPO significantly outperforms P-MDP baselines by 30%, achieving state-of-the-art performance on a widely-used benchmark, with notable advantages in problems requiring generalization. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary ORPO is a new way for computers to learn from experiences without actually trying them. It’s like a game where the computer makes decisions and gets rewards or penalties based on how well it does. The old way of doing this was too careful and didn’t take enough risks, which meant the computer didn’t get as good at making decisions. ORPO is different because it takes more risks and tries to learn from new situations. This helps the computer become better at making decisions in situations it hasn’t seen before. |
Keywords
* Artificial intelligence * Generalization * Optimization * Reinforcement learning
