Summary of Grid-mapping Pseudo-count Constraint For Offline Reinforcement Learning, by Yi Shen et al.
Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning
by Yi Shen, Hanyan Huang
First submitted to arxiv on: 3 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach to offline reinforcement learning, which learns from static datasets without interacting with environments. The method, called Grid-Mapping Pseudo-Count (GPC), extends count-based methods from discrete domains to continuous domains. GPC maps the continuous state and action space to a discrete grid, then constrains Q-values of out-of-distribution state-actions using pseudo-counts. Theoretical proofs show that GPC can achieve appropriate uncertainty constraints under fewer assumptions than other pseudo-count methods. When combined with Soft Actor-Critic (SAC), GPC-SAC algorithm is developed, which demonstrates better performance and lower computational cost compared to existing algorithms on D4RL datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about a new way for computers to learn from data without interacting with the world. It’s like trying to predict what will happen next based on patterns in old data. The problem is that the predictions might be wrong if we’re not careful, so this paper proposes a solution called Grid-Mapping Pseudo-Count (GPC). GPC helps by making sure the computer doesn’t get too confident or uncertain when it’s guessing what will happen. When combined with another learning method called Soft Actor-Critic, GPC-SAC shows that it can make better predictions and do so more efficiently. |
Keywords
* Artificial intelligence * Reinforcement learning
