Summary of Bridging State and History Representations: Understanding Self-predictive Rl, by Tianwei Ni et al.
Bridging State and History Representations: Understanding Self-Predictive RL
by Tianwei Ni, Benjamin Eysenbach, Erfan Seyedsalehi, Michel Ma, Clement Gehring, Aditya Mahajan, Pierre-Luc Bacon
First submitted to arxiv on: 17 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the commonalities among various representation learning methods in deep reinforcement learning (RL) for Markov decision processes (MDPs) and partially observable MDPs (POMDPs). Researchers have developed several techniques to understand effective representations, but the connections between these approaches remain unclear. The study reveals that many of these methods are based on a shared concept of self-predictive abstraction. Additionally, it provides insights into popular objectives and optimization strategies, such as stop-gradient techniques, for learning self-predictive representations. A minimalist algorithm is proposed to learn these representations, and its effectiveness is demonstrated through experiments on standard MDPs, MDPs with distractors, and POMDPs with sparse rewards. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how different methods in deep reinforcement learning (RL) are related. RL helps computers make decisions based on feedback. The researchers found that many of these methods use the same idea to create good representations. They also looked at why some methods work well and others don’t. This led them to develop a new way to learn representations, which they tested on different types of problems. |
Keywords
* Artificial intelligence * Optimization * Reinforcement learning * Representation learning
