Summary of Distributed Gradient Descent with Many Local Steps in Overparameterized Models, by Heng Zhu et al.
Distributed Gradient Descent with Many Local Steps in Overparameterized Models
by Heng Zhu, Harsh Vardhan, Arya Mazumdar
First submitted to arxiv on: 10 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, researchers investigate why the Federated Averaging (FedAvg) algorithm performs well in practice despite existing convergence analysis suggesting it degrades quickly with heterogeneous data. They analyze Local Gradient Descent (Local-GD) with a large number of local steps, showing that gradient descent at each node leads to implicit bias towards a specific direction locally. The paper characterizes the dynamics of the aggregated global model and compares it to the centralized model trained on all data. For linear regression tasks, the aggregated model converges exactly to the centralized model, while for classification tasks, it converges to the same feasible set as the centralized model. The authors also propose a Modified Local-GD algorithm that converges to the centralized model in direction for linear classification and verify their findings empirically on linear models and distributed fine-tuning of neural networks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how machine learning models are trained when data is shared among many devices. It’s about an important technique called Federated Averaging (FedAvg) that helps make sure all the devices work together well. The researchers want to know why FedAvg works better than expected, even when the data on each device is different. They found that the way each device updates its model creates a special kind of bias that helps the global model learn from all the data. This paper also proposes a new way to update models that’s even better than what’s currently used. |
Keywords
» Artificial intelligence » Classification » Fine tuning » Gradient descent » Linear regression » Machine learning
