Summary of Locmoe: a Low-overhead Moe For Large Language Model Training, by Jing Li et al.
LocMoE: A Low-Overhead MoE for Large Language Model Training
by Jing Li, Zhijie Sun, Xuan He, Li Zeng, Yi Lin, Entong Li, Binfan Zheng, Rongqian Zhao, Xin Chen
First submitted to arxiv on: 25 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The Mixtures-of-Experts (MoE) model is a popular method for distributed learning in large language models (LLMs), allowing for efficient sparsification and expansion. However, its performance is limited by load imbalance and high latency due to All-to-All communication and redundant computation resulting from large expert capacity. To address these issues, the authors propose a novel routing strategy that combines load balance and locality, converting partial inter-node communication to intra-node. They also identify a minimum threshold for expert capacity based on the maximal angular deviation between gating weights and assigned tokens. The proposed LocMoE is implemented on the PanGu-Sigma model using MindSpore framework with multi-level routing and tested on Ascend clusters, achieving 12.68% to 22.24% reduction in training time per epoch compared to classical routers without impacting accuracy. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The MoE model helps big language models learn better by sharing tasks between many smaller models. However, this approach has some problems like unequal workloads and slow communication between these models. The researchers suggest a new way of routing that balances the workload and reduces communication between models. They also find that there’s a minimum size for each small model to be effective. This new approach is tested on real-world data and shows it can train faster without losing accuracy. |
