Summary of On-device Collaborative Language Modeling Via a Mixture Of Generalists and Specialists, by Dongyang Fan et al.
On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists
by Dongyang Fan, Bettina Messmer, Nikita Doikov, Martin Jaggi
First submitted to arxiv on: 20 Sep 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach to federated learning, dubbed CoMiGS, which addresses two key challenges: computational resource heterogeneity and data heterogeneity among end users. The authors’ method, which combines mixture-of-experts learning with bi-level optimization, allows for the sharing of generalist experts across users while localizing specialist experts to adapt to individual resources and preserve privacy. The proposed approach is shown to effectively balance general and personalized knowledge in token generation, remaining robust against overfitting. The open-source codebase enables collaborative LLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper makes a new way to help devices learn together without sharing their data. This is important because it keeps users’ information private and lets devices with different abilities work together. The new method, CoMiGS, uses two types of experts: generalist and specialist. Generalists are shared among devices, while specialists are used on each device to fit its needs. The paper shows that this approach works well and is good at avoiding overfitting. Now, other researchers can use the code to make their own collaborative learning models. |
Keywords
» Artificial intelligence » Federated learning » Mixture of experts » Optimization » Overfitting » Token
