Summary of Upcycling Instruction Tuning From Dense to Mixture-of-experts Via Parameter Merging, by Tingfeng Hui et al.
Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
by Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Yu Sun, Hua Wu, Sen Su
First submitted to arxiv on: 2 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a novel approach called Upcycling Instruction Tuning (UpIT) to transform large language models (LLMs) from dense to Mixture-of-Experts (MoE) models. Existing methods require significant data and rely on post-training, whereas UpIT is data-efficient. The approach involves expanding intermediate checkpoints during instruction tuning into MoE models using genetic algorithms and parameter merging. To ensure each expert functions correctly, a small seed dataset is used to pre-optimize the router. Experiments demonstrate UpIT’s outstanding performance and data efficiency, as well as stable improvement in scaling experts or data. The importance of ensuring expert diversity in upcycling is also highlighted. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper shows how to make big language models better by breaking them into smaller teams that work together. Right now, making these models work well requires a lot of data and tweaking after they’re trained. But the new approach, called UpIT, can do it with much less data. It works by taking snapshots of the model during training and then adding more “experts” to help it learn specific tasks. The experts are pre-trained on small amounts of data before being added to the model. This helps them work well together and makes the whole model better at understanding language. |
Keywords
» Artificial intelligence » Instruction tuning » Mixture of experts
