Summary of Adaptive Exploration For Data-efficient General Value Function Evaluations, by Arushi Jain et al.
Adaptive Exploration for Data-Efficient General Value Function Evaluations
by Arushi Jain, Josiah P. Hanna, Doina Precup
First submitted to arxiv on: 13 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach to learning General Value Functions (GVFs) in reinforcement learning is introduced. Existing methods face efficiency issues when learning multiple GVFs using off-policy methods. The GVFExplorer method adaptively learns a single behavior policy that efficiently collects data for evaluating multiple GVFs, optimizing the policy by minimizing the total variance in return across GVFs. The approach uses an existing temporal-difference-style variance estimator to approximate the return variance and proves that each behavior policy update decreases the overall mean squared error in GVF predictions. Empirical results demonstrate the method’s performance in tabular and nonlinear function approximation settings, including Mujoco environments with stationary and non-stationary reward signals. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary GVFs represent predictive knowledge in reinforcement learning. The new approach, called GVFExplorer, helps learn multiple GVFs more efficiently using off-policy methods. It does this by finding a single behavior policy that collects the right data to evaluate all the GVFs at once. This is done by minimizing the total variance in return across GVFs. The method uses an existing way to estimate return variance and shows that each update of the policy makes predictions better for all GVFs. The results show that this approach works well in different environments. |
Keywords
» Artificial intelligence » Reinforcement learning
