Summary of Fedbiot: Llm Local Fine-tuning in Federated Learning Without Full Model, by Feijie Wu et al.
FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model
by Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, Jing Gao
First submitted to arxiv on: 25 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach to fine-tuning large language models (LLMs) in federated learning (FL), called FedBiOT. The authors address the challenge of LLM fine-tuning on privately distributed domain-specific data, leveraging limited computation and communication capacities of FL clients. Their method involves the server generating a compressed LLM and aligning its performance with the full model, while clients fine-tune an adapter, a lightweight yet important part of the compressed model. The authors formulate this problem as a bi-level optimization problem to minimize the negative effect of data discrepancy and derive updating rules for the server and clients. They conduct extensive experiments on LLaMA-2, demonstrating exceptional performance when the adapter is reintegrated into the global LLM. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps us learn how to make language models better at understanding specific topics like medicine or science, even though we can’t share all our data with each other. The problem is that we have lots of important information hidden away on different computers, and we don’t want to send it all over the place because that would be too slow or take up too much space. So, they came up with a clever way called FedBiOT to help these computers learn from each other without sharing their data. They showed that this approach works well by testing it on a big language model and getting good results. |
Keywords
» Artificial intelligence » Federated learning » Fine tuning » Language model » Llama » Optimization
