Summary of Dual Low-rank Adaptation For Continual Learning with Pre-trained Models, by Huancheng Chen and Jingtao Li and Nidham Gazagnadou and Weiming Zhuang and Chen Chen and Lingjuan Lyu
Dual Low-Rank Adaptation for Continual Learning with Pre-Trained Models
by Huancheng Chen, Jingtao Li, Nidham Gazagnadou, Weiming Zhuang, Chen Chen, Lingjuan Lyu
First submitted to arxiv on: 1 Nov 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel continual learning method called Dual Low-Rank Adaptation (DualLoRA) is proposed to enable vision transformers (ViTs) to learn new tasks over time. This approach combines the benefits of parameter-efficient fine-tuning (PEFT) and LoRA, with a dynamic memory mechanism that balances stability and plasticity. The method consists of an orthogonal LoRA adapter that updates parameters in an orthogonal subspace of previous tasks to mitigate catastrophic forgetting, and a residual LoRA adapter that updates parameters in the residual subspace spanned by task-specific bases without interaction across tasks. This combination allows for faster convergence, fewer trainable parameters, and improved accuracy, inference speed, and memory efficiency compared to existing CL methods on multiple benchmarks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Continual learning helps machines learn new things over time. Researchers are working on ways to make this happen with big models called vision transformers (ViTs). One problem is that these models forget what they learned before when they’re learning something new. A new method called DualLoRA tries to solve this by using two special adapters. The first adapter helps the model remember what it learned before, while the second adapter lets the model learn new things without forgetting too much. This makes the model better at doing tasks and uses less memory. |
Keywords
* Artificial intelligence * Continual learning * Fine tuning * Inference * Lora * Low rank adaptation * Parameter efficient
