Summary of Ferret: Federated Full-parameter Tuning at Scale For Large Language Models, by Yao Shu et al.
Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
by Yao Shu, Wenyang Hu, See-Kiong Ng, Bryan Kian Hsiang Low, Fei Richard Yu
First submitted to arxiv on: 10 Sep 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach to fine-tuning Large Language Models (LLMs) in federated settings is proposed, addressing the trade-off between communication efficiency and model accuracy. The first-order method, Ferret, enables scalable full-parameter tuning of LLMs across decentralized data sources while maintaining competitive model accuracy. This is achieved through efficient local updates, projection into a low-dimensional space to reduce communication overhead, and reconstruction of local updates with shared randomness for global aggregation. Experimental results demonstrate that Ferret enhances the scalability of existing federated full-parameter tuning approaches, achieving high computational efficiency, reduced communication overhead, and fast convergence while maintaining competitive model accuracy. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Federated learning is a way to train AI models using data from many different places. This can be hard because each place might have different rules about sharing data. Researchers developed a new method called Ferret that makes it easier to train language models in this kind of setting. It works by taking small steps to update the model, then combining those updates with others from other places. This helps keep the model accurate and efficient. The results show that Ferret is better than previous methods at balancing these two goals. |
Keywords
» Artificial intelligence » Federated learning » Fine tuning
