Summary of Analyzing and Bridging the Gap Between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning, by Shuyu Yin et al.
Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning
by Shuyu Yin, Fei Wen, Peilin Liu, Tao Luo
First submitted to arxiv on: 18 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the optimal objective in reinforcement learning (RL), a crucial aspect determining policy evaluation and optimization. While total return maximization is ideal, discounted return maximization is practical due to stability concerns. This misalignment can lead to suboptimal policies. The authors analyze the performance gap between total return and discounted return maximizing policies, revealing that increasing discount factors may not eliminate this gap in environments with cyclic states. To address this issue, they propose two alternative approaches: modifying terminal state values as tunable hyperparameters or calibrating reward data in trajectories using off-policy algorithms. These methods enhance robustness to the discount factor and improve performance in large trajectory lengths. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how we measure success in machine learning systems that make decisions based on rewards. Right now, we use a method called discounted return maximization, but this can lead to problems if the system doesn’t always get the reward it wants. The authors investigate why this is happening and propose two new ways to solve the problem: one adjusts how we value the end of an episode, and another adjusts the rewards themselves. This could help make machine learning systems more robust and better at making decisions. |
Keywords
» Artificial intelligence » Machine learning » Optimization » Reinforcement learning
