Summary of Feduv: Uniformity and Variance For Heterogeneous Federated Learning, by Ha Min Son et al.
FedUV: Uniformity and Variance for Heterogeneous Federated Learning
by Ha Min Son, Moon-Hyun Kim, Tai-Myoung Chung, Chao Huang, Xin Liu
First submitted to arxiv on: 27 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, researchers investigate the training dynamics of neural networks trained with federated learning, where data is distributed heterogeneously. They propose two regularization terms to mitigate local bias by making local models behave as if they were in an IID setting. The authors apply Singular Value Decomposition (SVD) to the weights and find that there are differences between IID and non-IID settings. They then introduce variance regularization in dimension-wise probability distributions and hyperspherical uniformity of representations, promoting local models to act as if they were in an IID setting regardless of local data distribution. The proposed method achieves highest performance by a large margin, especially in highly non-IID cases, and is scalable to larger models and datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Federated learning helps train neural networks with widely spread data, but this can be tricky when the data isn’t mixed together. Some people have tried freezing the last layer of the network, which seems to help. The researchers in this paper look into why that works. They think it’s because the final layer is super sensitive to local biases in the data. To fix this, they create two new tricks: one makes sure each piece of data looks like it came from the same place, and another makes sure all the data points are equally spread out. This helps train the networks better, especially when the data is really mixed up. |
Keywords
* Artificial intelligence * Federated learning * Probability * Regularization
