Summary of Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models, by Ji Liu et al.
Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models
by Ji Liu, Jiaxiang Ren, Ruoming Jin, Zijie Zhang, Yang Zhou, Patrick Valduriez, Dejing Dou
First submitted to arxiv on: 30 Sep 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Federated Learning (FL) is a promising approach for training Large Language Models (LLMs) collaboratively with decentralized data. While LLMs have huge sizes, the scale of training data increases significantly, resulting in tremendous computation and communication costs. The non-Independent and Identically Distributed (non-IID) nature of the training data requires adaptive processing within each device. To address this challenge, we propose a Fisher Information-based Efficient Curriculum Federated Learning framework (FibecFed), which includes two novel methods: adaptive federated curriculum learning and efficient sparse parameter update. Our approach adaptively samples data within each device to improve fine-tuning effectiveness and dynamically selects layers for global aggregation and sparse parameters for local update with LoRA to enhance efficiency. Extensive experiments on 10 datasets demonstrate that FibecFed achieves excellent performance (up to 45.35% in terms of accuracy) and superior fine-tuning speed (up to 98.61% faster) compared to 17 baseline approaches. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine being able to train powerful language models using data from many different places, all at once. This is the idea behind Federated Learning (FL). The problem is that these models need a lot of training data, which can be hard to get and process. To make this process more efficient, we propose a new approach called FibecFed. It uses two main ideas: sampling data in a smart way and updating model parameters efficiently. We tested our approach on 10 different datasets and found that it works better and faster than many other methods. |
Keywords
» Artificial intelligence » Curriculum learning » Federated learning » Fine tuning » Lora
