Summary of Activations and Gradients Compression For Model-parallel Training, by Mikhail Rudakov et al.
Activations and Gradients Compression for Model-Parallel Training
by Mikhail Rudakov, Aleksandr Beznosikov, Yaroslav Kholodov, Alexander Gasnikov
First submitted to arxiv on: 15 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Distributed, Parallel, and Cluster Computing (cs.DC); Optimization and Control (math.OC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper explores ways to improve model-parallel training of large neural networks, which often rely on distributed computing systems. The authors investigate how compressing both activations and gradients affects the convergence of these models during training. They examine various compression methods, including quantization and TopK compression, as well as error compensation techniques like per-batch error feedback. Experiments are conducted on image classification and language model fine-tuning tasks to determine the optimal compression rates for different components of the model. The findings suggest that gradients require more gentle compression than activations, while applying TopK compression with AQ-SGD during training can lead to improved performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large neural networks need many computers to train. One way to speed up this process is by splitting the model between computers and compressing the information they share. This helps reduce the time it takes for them to communicate. The researchers looked at different ways to compress this information, like reducing the precision of calculations or keeping only the most important parts. They also tried adding error feedback to help the models learn better. The experiments were done on image classification and language model fine-tuning tasks. The results show that gradients need less compression than activations, and applying TopK compression with AQ-SGD during training can improve performance. |
Keywords
* Artificial intelligence * Fine tuning * Image classification * Language model * Precision * Quantization
