Summary of A Unifying Framework For Action-conditional Self-predictive Reinforcement Learning, by Khimya Khetarpal et al.
A Unifying Framework for Action-Conditional Self-Predictive Reinforcement Learning
by Khimya Khetarpal, Zhaohan Daniel Guo, Bernardo Avila Pires, Yunhao Tang, Clare Lyle, Mark Rowland, Nicolas Heess, Diana Borsa, Arthur Guez, Will Dabney
First submitted to arxiv on: 4 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates reinforcement learning (RL) agents’ ability to learn good representations by bootstrapping from future latent representations using the self-predictive learning algorithm. Building on previous work, which focused on a simplified continuous-time ODE model for self-predictive representation learning under a fixed policy assumption, this study analyzes an action-conditional self-predictive objective (BYOL-AC) and its convergence properties. The authors show how BYOL-AC is related to the variance equation and introduce a novel variance-like action-conditional objective (BYOL-VAR). They also unify the study of all three objectives through model-based and model-free perspectives, demonstrating that BYOL-AC outperforms other objectives in various RL environments. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how machines can learn to make good decisions by using information from their own predictions. It builds on earlier work that showed how a simple mathematical model could help machines learn representations for decision-making tasks. In this study, the researchers analyze a more complex version of this model and show how it relates to a measure of uncertainty called variance. They also introduce a new way to use this model and demonstrate its performance in different scenarios. |
Keywords
» Artificial intelligence » Bootstrapping » Reinforcement learning » Representation learning
