Summary of Remembering Transformer For Continual Learning, by Yuwei Sun et al.
Remembering Transformer for Continual Learning
by Yuwei Sun, Ippei Fujisawa, Arthur Juliani, Jun Sakuma, Ryota Kanai
First submitted to arxiv on: 11 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The Remembering Transformer addresses the challenge of catastrophic forgetting in continual learning, where new task learning interferes with previously learned knowledge. It proposes a mixture-of-adapters architecture and novelty detection mechanism within a pretrained Transformer to alleviate this issue. The model dynamically routes task data to the most relevant adapter based on knowledge distillation, achieving better parameter efficiency. Experimental results demonstrate state-of-the-art performance in various class-incremental split tasks and permutation tasks, outperforming the second-best method by 15.90%. This approach also reduces the memory footprint from 11.18M to 0.22M in the five splits CIFAR10 task. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The Remembering Transformer is a new way for computers to learn and remember things without forgetting what they already know. It’s like a special kind of memory that helps machines learn more and more without losing what they learned before. The model uses a special trick called “mixture-of-adapters” to make this happen, which allows it to adapt to new tasks without forgetting old ones. This is important because it means machines can keep learning and improving over time, just like humans do. |
Keywords
» Artificial intelligence » Continual learning » Knowledge distillation » Novelty detection » Transformer
