Summary of Heavy-ball Momentum Accelerated Actor-critic with Function Approximation, by Yanjie Dong et al.
Heavy-Ball Momentum Accelerated Actor-Critic With Function Approximation
by Yanjie Dong, Haijun Zhang, Gang Wang, Shisheng Cui, Xiping Hu
First submitted to arxiv on: 13 Aug 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed heavy-ball momentum based advantage actor-critic (HB-A2C) algorithm integrates a linear function into the critic recursion, reducing the variance of stochastic policy gradient and improving convergence rate. By analyzing the impact of momentum on AC algorithms under Markovian noise, this work demonstrates that HB-A2C finds an ε-approximate stationary point with O(ε^(-2)) iterations for reinforcement learning tasks. Additionally, it reveals the dependence of learning rates on sample trajectory length, allowing for careful selection of momentum factors to balance initialization and stochastic approximation errors. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper proposes a new way to improve actor-critic algorithms by using “heavy-ball” momentum. This helps reduce the uncertainty in the algorithm’s updates, making it work better. The authors show that their method can find good solutions to reinforcement learning problems quickly, and they also explain how to choose the right amount of momentum to balance out the errors. |
Keywords
* Artificial intelligence * Reinforcement learning
