Summary of Convergence Of Distributed Adaptive Optimization with Local Updates, by Ziheng Cheng et al.
Convergence of Distributed Adaptive Optimization with Local Updates
by Ziheng Cheng, Margalit Glasgow
First submitted to arxiv on: 20 Sep 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Optimization and Control (math.OC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates distributed adaptive algorithms with local updates, focusing on reducing communication complexity in machine learning model training. The authors study Local SGD with momentum (Local SGDM) and Adam (Local Adam), showing that these methods can outperform their minibatch counterparts in certain regimes for convex and weakly convex problems. The analysis relies on a novel technique to prove contraction during local iterations, under generalized smoothness assumptions and gradient clipping strategy. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Distributed adaptive algorithms are very useful for training machine learning models, but until now, it wasn’t clear how they could benefit from letting different parts of the network do their own updates. This paper shows that some versions of these algorithms can actually be better than others at solving certain types of problems. The authors use a new way to prove that this is true, which involves looking at what happens when each part of the network does its own update. |
Keywords
* Artificial intelligence * Machine learning
