Summary of Pipeline Gradient-based Model Training on Analog In-memory Accelerators, by Zhaoxian Wu et al.
Pipeline Gradient-based Model Training on Analog In-memory Accelerators
by Zhaoxian Wu, Quan Xiao, Tayfun Gokmen, Hsinyu Tsai, Kaoutar El Maghraoui, Tianyi Chen
First submitted to arxiv on: 19 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Hardware Architecture (cs.AR); Optimization and Control (math.OC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Analog In-Memory Computing (AIMC) accelerator aims to accelerate the training of large deep neural models in an energy-efficient way. AIMC accelerators store trainable weights in memory, reducing overhead by eliminating the need for weight movement between memory and processors during training. However, this feature constrains data parallelism due to expensive weight copying between AIMCs. To enable parallel training, pipeline parallelism is proposed for AIMC accelerators, inspired by digital domain pipelines. This paper provides theoretical convergence guarantees for synchronous and asynchronous pipelines in terms of sampling and clock cycle complexity, considering the asymmetric bias caused by physical AIMC accelerator updates. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Analog computers are trying to make deep learning faster and more energy-efficient. They store weights in memory instead of moving them between different parts of the computer during training. This helps, but it also makes it harder to train models using many processors at once. To fix this problem, researchers propose a new way to divide up the training process into smaller steps that can be done in parallel on multiple processors. They show that this approach works well for deep neural networks and is more energy-efficient than traditional methods. |
Keywords
» Artificial intelligence » Deep learning
