Summary of Revisiting Smoe Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning, By Soumajyoti Sarkar et al.
Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
by Soumajyoti Sarkar, Leonard Lausen, Volkan Cevher, Sheng Zha, Thomas Brox, George Karypis
First submitted to arxiv on: 2 Sep 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a novel approach to designing Sparse Mixture of Expert (SMoE) models for language modeling. By leveraging conditionally activated feedforward subnetworks in transformer blocks, SMoE models offer a scalable alternative to dense models. However, the authors identify a challenge with large token-routed SMoE models: during inference, the entire model must be used, resulting in high latencies in distributed settings. To address this issue, the researchers introduce an adaptive task-aware pruning technique called UNCURL to reduce the number of experts per MoE layer post-training. The findings reveal a threshold pruning factor that depends on the number of experts used in pretraining, above which the reduction degrades model performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how to design better language models using something called Sparse Mixture of Expert (SMoE) models. These models are good because they can handle lots of data without getting too slow. But big SMoE models have a problem: when we need to use them, the whole model has to be used, which takes a long time. The researchers found a way to make these models smaller and faster using a special technique called UNCURL. They also figured out that there’s a limit to how much you can shrink the model before it starts getting worse. |
Keywords
» Artificial intelligence » Inference » Pretraining » Pruning » Token » Transformer
