Summary of Communication-efficient Federated Group Distributionally Robust Optimization, by Zhishuai Guo et al.
Communication-Efficient Federated Group Distributionally Robust Optimization
by Zhishuai Guo, Tianbao Yang
First submitted to arxiv on: 8 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper tackles the challenges faced by federated learning due to data heterogeneity at different clients, which can compromise model generalization ability. Existing approaches based on group distributionally robust optimization (GDRO) often lead to high communication and sample complexity. To address this issue, the authors introduce algorithms for communication-efficient Federated Group Distributionally Robust Optimization (FGDRO). The contributions are threefold: FGDRO-CVaR optimizes average top-K losses while reducing communication complexity; FGDRO-KL optimizes KL regularized FGDRO with reduced communication complexity; and FGDRO-KL-Adam combines Adam-type local updates in FGDRO-KL, maintaining a communication cost. The algorithms are demonstrated to be effective on real-world tasks like natural language processing and computer vision. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Federated learning is a way for different devices or clients to work together using machine learning models. But there’s a problem: the data each client has is often very different, which can make it hard for the model to work well in many situations. Researchers have tried to fix this with group distributionally robust optimization (GDRO), but that approach has its own problems – it requires a lot of communication and data. In this paper, the authors introduce new algorithms that make GDRO more efficient and effective. They show how their methods can be used for tasks like language processing and computer vision. |
Keywords
» Artificial intelligence » Federated learning » Generalization » Machine learning » Natural language processing » Optimization
