Summary of Decision Mamba: Reinforcement Learning Via Hybrid Selective Sequence Modeling, by Sili Huang et al.
Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling
by Sili Huang, Jifeng Hu, Zhejian Yang, Liwei Yang, Tao Luo, Hechang Chen, Lichao Sun, Bo Yang
First submitted to arxiv on: 31 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Recent advancements in transformer models have led to remarkable successes in reinforcement learning (RL), particularly when applied to sequential generation. The concept of in-context RL, which involves providing task contexts such as multiple trajectories, has emerged as a promising approach. However, current methods suffer from high computational costs due to the quadratic complexity of attention mechanisms. This limitation hinders the ability to tackle tasks requiring long-term memory. The Mamba model, known for its efficient processing of long-term dependencies, offers an opportunity for in-context RL to overcome this challenge. This paper proposes Decision Mamba (DM) and Decision Mamba-Hybrid (DM-H), which leverage the strengths of transformers and Mamba in high-quality prediction and long-term memory. DM-H first generates sub-goals from long-term memory using the Mamba model, then uses these sub-goals to prompt a transformer for high-quality predictions. Experimental results demonstrate state-of-the-art performance on tasks such as D4RL, Grid World, and Tmaze benchmarks, with efficiency improvements of up to 28 times compared to transformer-based baselines. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you’re trying to learn from experience, like a robot playing a game. Researchers have found that special computer models called transformers are really good at this kind of learning. They can make decisions based on what happened before. However, these models get stuck when they need to remember things from long ago. A new approach called in-context RL tries to solve this problem by giving the model more information about the task it’s trying to accomplish. This paper proposes a way to combine the strengths of transformers and another special model called Mamba to make better decisions and remember things for longer. The results show that this approach is really effective and can even do tasks much faster than before. |
Keywords
» Artificial intelligence » Attention » Prompt » Reinforcement learning » Transformer
