Summary of Fedmcp: Parameter-efficient Federated Learning with Model-contrastive Personalization, by Qianyi Zhao et al.
FedMCP: Parameter-Efficient Federated Learning with Model-Contrastive Personalization
by Qianyi Zhao, Chen Qu, Cen Chen, Mingyuan Fan, Yanhao Wang
First submitted to arxiv on: 28 Aug 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach called FedMCP to fine-tune pre-trained language models in federated learning. The existing methods face two primary challenges: excessive communication and computational overhead due to the large number of parameters, and heterogeneous data and tasks across clients. To address these issues, FedMCP adds two lightweight adapter modules to frozen PLMs within clients, which are then aggregated at the server level. Additionally, a model-contrastive regularization term is introduced between the adapters to encourage universal knowledge assimilation and client-specific knowledge capture. Experimental results on highly heterogeneous datasets show that FedMCP achieves substantial performance improvements over state-of-the-art FL fine-tuning approaches for PLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making sure computers can learn from lots of different data without sharing all the data itself. This is important because some data might be private or sensitive, and it’s hard to make sure that everyone is learning equally well from their own data. The researchers came up with a new way called FedMCP to help computers learn better by adding special “adapter” modules to the existing models. These adapters help the computer understand what’s important in each piece of data, so it can use that knowledge to improve its performance on different tasks. |
Keywords
» Artificial intelligence » Federated learning » Fine tuning » Regularization
