Summary of Skill-aware Mutual Information Optimisation For Generalisation in Reinforcement Learning, by Xuehui Yu et al.
Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learning
by Xuehui Yu, Mhairi Dunion, Xin Li, Stefano V. Albrecht
First submitted to arxiv on: 7 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Robotics (cs.RO)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Meta-Reinforcement Learning (Meta-RL) agents struggle with varying environmental features requiring different optimal skills. To improve generalisation, we introduce Skill-aware Mutual Information (SaMI), an optimisation objective distinguishing context embeddings according to skills, enabling RL agents to identify and execute different skills across tasks. We also propose Skill-aware Noise Contrastive Estimation (SaNCE), a K-sample estimator used to optimise SaMI. Our framework equips RL agents with SaNCE in practice, validated on modified MuJoCo and Panda-gym benchmarks. Agents maximising SaMI achieve improved zero-shot generalisation, and the context encoder trained with SaNCE shows robustness to reduced sample sizes, potentially overcoming the log-K curse. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Meta-Reinforcement Learning (ML) agents have trouble working across different tasks that require different skills. A new way to help ML agents learn from this experience is by using something called Skill-aware Mutual Information (SaMI). SaMI helps the agent understand what skill it needs to use in a particular situation. We also came up with an idea called Skill-aware Noise Contrastive Estimation (SaNCE), which makes sure the agent can use SaMI effectively. Our new approach, called SaNCE, is tested on some special computer simulations and shows that agents using SaMI can learn faster and better than before. |
Keywords
* Artificial intelligence * Encoder * Reinforcement learning * Zero shot
