Summary of Efficient Backpropagation with Variance-controlled Adaptive Sampling, by Ziteng Wang et al.
Efficient Backpropagation with Variance-Controlled Adaptive Sampling
by Ziteng Wang, Jianfei Chen, Jun Zhu
First submitted to arxiv on: 27 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel neural network training algorithm, variance-controlled adaptive sampling (VCAS), is introduced to accelerate backpropagation (BP) while maintaining accuracy across various tasks. VCAS uses importance sampling and leverage score sampling to reduce computations during activation gradient calculation and weight gradient calculation, respectively. The method learns the sample ratio jointly with model parameters to control additional variance and preserve training loss trajectory and validation accuracy. Experimental results show that VCAS achieves up to 73.87% FLOPs reduction of BP and 49.58% FLOPs reduction of the whole training process on multiple fine-tuning and pre-training tasks in vision and natural language domains. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Neural networks are powerful tools for artificial intelligence, but they can be slow to train. A new way to speed up training is called variance-controlled adaptive sampling (VCAS). VCAS works by skipping some of the calculations that aren’t very important. It does this by learning how to sample data and weights in a smart way. This helps keep the results accurate while making the training process faster. In tests, VCAS was able to speed up training by up to 73% without losing any accuracy. |
Keywords
* Artificial intelligence * Backpropagation * Fine tuning * Neural network
