Summary of Constrained Latent Action Policies For Model-based Offline Reinforcement Learning, by Marvin Alles et al.
Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning
by Marvin Alles, Philip Becker-Ehmck, Patrick van der Smagt, Maximilian Karl
First submitted to arxiv on: 7 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Constrained Latent Action Policies (C-LAP) method learns a generative model of joint observations and actions in offline reinforcement learning. This approach casts policy learning as a constrained objective to stay within the support of the latent action distribution, eliminating the need for uncertainty penalties on the Bellman update. The method uses the generative capabilities of the model to impose an implicit constraint on generated actions. C-LAP is evaluated on D4RL and V-D4RL benchmark datasets, showing competitive performance with state-of-the-art methods, particularly in datasets with visual observations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Offline reinforcement learning learns policies using static datasets without environment feedback. This setting poses challenges like generating out-of-distribution samples. Model-based methods try to overcome these by learning environment dynamics and guiding policy search. Current methods use conservatism on the Bellman update, often via uncertainty estimation from model ensembles. The proposed C-LAP method learns a joint observation-action generative model, casting policy learning as a constrained objective to stay within the support of the latent action distribution. This eliminates the need for additional uncertainty penalties and reduces gradient steps required to learn a policy. |
Keywords
» Artificial intelligence » Generative model » Reinforcement learning
