Summary of Trustworthiness Of Stochastic Gradient Descent in Distributed Learning, by Hongyang Li et al.
Trustworthiness of Stochastic Gradient Descent in Distributed Learning
by Hongyang Li, Caesar Wu, Mohammed Chadli, Said Mammar, Pascal Bouvry
First submitted to arxiv on: 28 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper explores the intersection of distributed learning and optimization algorithms. Specifically, it investigates the challenges posed by communication bottlenecks in large-scale model training using Stochastic Gradient Descent (SGD). To mitigate these issues, researchers have developed compressed SGD techniques, which reduce the amount of data exchanged between nodes. However, this approach introduces new concerns about trustworthiness, as gradient exchanges are vulnerable to attacks like gradient inversion and membership inference attacks. The paper aims to fill a gap in understanding the reliability of compressed SGD under such threats. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Distributed learning uses many computers to train models quickly and efficiently. One important part of this process is an algorithm called Stochastic Gradient Descent (SGD). While it helps, there’s a problem – when all those computers talk to each other, it can slow things down. To fix this, scientists developed a way to send less information between the computers. But now, there are new worries about how trustworthy this process is. Cyberattacks could mess with the training data, making it hard to know what the model really learned. This paper looks into these concerns and tries to answer important questions about how reliable this process is. |
Keywords
» Artificial intelligence » Inference » Optimization » Stochastic gradient descent
