Summary of Roer: Regularized Optimal Experience Replay, by Changling Li et al.
ROER: Regularized Optimal Experience Replay
by Changling Li, Zhang-Wei Hong, Pulkit Agrawal, Divyansh Garg, Joni Pajarinen
First submitted to arxiv on: 4 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Robotics (cs.RO)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach to prioritizing experiences in online reinforcement learning (RL) using the temporal difference (TD) error. The authors provide an alternative perspective on TD-error-based reweighting and show connections between experience prioritization and occupancy optimization. They introduce a regularized RL objective with an f-divergence regularizer and derive an optimal solution that shifts the distribution of off-policy data in the replay buffer towards the on-policy optimal distribution using TD-error-based occupancy ratios. The proposed prioritization scheme, Regularized Optimal Experience Replay (ROER), is evaluated with the Soft Actor-Critic (SAC) algorithm in continuous control MuJoCo and DM Control benchmark tasks, outperforming baselines in 6 out of 11 tasks. Moreover, ROER achieves noticeable improvement on the Antmaze environment when used for offline-to-online fine-tuning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper finds a new way to help computers learn from past experiences while making decisions. The researchers explore how to prioritize these experiences based on how well they match what the computer is trying to achieve. They come up with a new approach that uses this prioritization to make better choices. This helps the computer learn faster and more accurately in certain situations. |
Keywords
* Artificial intelligence * Fine tuning * Optimization * Reinforcement learning
