Summary of Stable Offline Value Function Learning with Bisimulation-based Representations, by Brahma S. Pavse et al.
Stable Offline Value Function Learning with Bisimulation-based Representations
by Brahma S. Pavse, Yudong Chen, Qiaomin Xie, Josiah P. Hanna
First submitted to arxiv on: 2 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the stability of offline value function learning in reinforcement learning, specifically the procedure of using an offline dataset to estimate the expected discounted return from each state when taking actions according to a fixed target policy. The authors highlight that poorly learned representations can make value function learning unstable or divergent, and thus introduce a bisimulation-based algorithm called kernel representations for offline policy evaluation (KROPE) to stabilize value function learning. KROPE uses a kernel to shape state-action representations such that similar immediate rewards and next state-action pairs under the target policy are represented similarly. The authors demonstrate that KROPE learns stable representations and achieves lower value error than baselines. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary In this paper, researchers study how to make a process in reinforcement learning more reliable. They look at ways to learn about good actions to take in different situations using data collected before. Sometimes, this process can get stuck or go wrong if the way we represent states and actions isn’t good enough. To fix this, they create an algorithm that helps shape these representations so they are useful for making decisions. This new algorithm is called kernel representations for offline policy evaluation (KROPE). The researchers show that KROPE does a better job than other methods at learning about what actions to take in different situations. |
Keywords
* Artificial intelligence * Reinforcement learning
