Summary of Reducing Fine-tuning Memory Overhead by Approximate and Memory-sharing Backpropagation, By Yuchen Yang et al.
Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation
by Yuchen Yang, Yingdong Shi, Cheems Wang, Xiantong Zhen, Yuxuan Shi, Jun Xu
First submitted to arxiv on: 24 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper proposes innovative methods to reduce memory overhead in fine-tuning large-scale models for downstream tasks. The authors focus on the activation function and layer normalization perspectives, developing the Approximate Backpropagation (Approx-BP) theory to decouple forward and backward passes. This leads to memory-efficient alternatives of GELU and SiLU activation functions using derivative functions of ReLUs in the backward pass. Additionally, the Memory-Sharing Backpropagation strategy is introduced, allowing for sharing activation memory between adjacent layers, reducing redundancy. The proposed methods do not incur extra computation or reduce training efficiency. Extensive experiments with vision and language models demonstrate a peak memory usage reduction of up to 30%. The code is released publicly. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research helps make big computer models more efficient by using less memory. The scientists developed new ways to calculate the model’s output and used tricks to share information between layers, reducing the amount of memory needed. They tested their methods on popular computer vision and language models and showed that they can cut memory usage in half! This is important because it means we can train even bigger and more powerful models without running out of memory. |
Keywords
» Artificial intelligence » Backpropagation » Fine tuning
