Summary of Routers in Vision Mixture Of Experts: An Empirical Study, by Tianlin Liu et al.
Routers in Vision Mixture of Experts: An Empirical Study
by Tianlin Liu, Mathieu Blondel, Carlos Riquelme, Joan Puigcerver
First submitted to arxiv on: 29 Jan 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This comprehensive study delves into Mixture-of-Experts (MoE) models for computer vision tasks, focusing on the router’s pivotal role in determining which subset of parameters processes feature embeddings. The unified MoE formulation subsumes various MoEs with two parametric routing tensors, encompassing sparse and soft MoEs. The research explores six routers, including existing ones from prior work and new ones introduced, demonstrating that many language model-inspired routers can be adapted for strong performance in vision tasks. Key findings reveal that Expert Choice routers generally outperform Token Choice routers in sparse MoE, while soft MoEs consistently surpass sparse MoEs with a fixed compute budget. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study explores how to make Mixture-of-Experts (MoE) models better for computer vision. MOEs are like super powerful computers that can do lots of things at once. A key part of MOEs is the “router” – it decides which parts of the computer work on different things. The researchers looked at many different routers and found some work better than others. They also showed that some types of MOEs, called soft MOEs, are generally better than others. |
Keywords
* Artificial intelligence * Language model * Mixture of experts * Token
