Summary of Watch Your Head: Assembling Projection Heads to Save the Reliability Of Federated Models, by Jinqian Chen et al.
Watch Your Head: Assembling Projection Heads to Save the Reliability of Federated Models
by Jinqian Chen, Jihua Zhu, Qinghai Zheng, Zhongyu Li, Zhiqiang Tian
First submitted to arxiv on: 26 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, researchers tackle the issue of unreliable federated learning when dealing with heterogeneous data. Despite progress made in mitigating performance degradation and convergence issues, the reliability aspect of federated models has been largely overlooked. The study investigates both generic and personalized federated models, revealing that they exhibit unreliability when faced with heterogeneous data, leading to poor calibration on in-distribution test data and low uncertainty levels on out-of-distribution data. This is primarily attributed to biased projection heads introducing miscalibration into the models. To address this, the authors propose the Assembled Projection Heads (APH) method, which randomly samples multiple initialized parameters of projection heads from a prior and fine-tunes them on locally available data under varying learning rates. This head ensemble introduces parameter diversity, eliminating bias and producing reliable predictions via head averaging. The proposed APH method is evaluated across three prominent federated benchmarks, demonstrating its efficacy in model calibration and uncertainty estimation. Notably, APH can be seamlessly integrated into various federated approaches with only a 30% additional computation cost for 100x inferences within large models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Federated learning helps different devices learn together without sharing their data directly. But when the data is very different between devices, this approach doesn’t work well and the results are not reliable. The researchers studied this problem and found that it’s because of biased “projections” in the model, which make it unreliable. They proposed a new method called Assembled Projection Heads (APH) to fix this issue. APH works by using many different projections and combining their predictions to get a more accurate result. The study shows that APH makes the models more reliable and can be used with most existing federated learning approaches. |
Keywords
* Artificial intelligence * Federated learning
