Summary of Lora-switch: Boosting the Efficiency Of Dynamic Llm Adapters Via System-algorithm Co-design, by Rui Kong et al.
LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design
by Rui Kong, Qiyang Li, Xinyu Fang, Qingtian Feng, Qingfeng He, Yazhu Dong, Weijun Wang, Yuanchun Li, Linghe Kong, Yunxin Liu
First submitted to arxiv on: 28 May 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates ways to improve large language models (LLMs) by adding dynamic adapters, such as low-rank adapters (LoRA) with Mixture-of-Experts (MoE) structures. While these adapters offer modest computational complexity, they surprisingly incur huge inference latency overhead, slowing down decoding speed by 2.5+ times. The authors analyze the fine-grained costs of these adapters and identify fragmented CUDA kernel calls as the root cause. To address this, they propose LoRA-Switch, a system-algorithm co-designed architecture for efficient dynamic adapters. Unlike existing approaches that adopt layer-wise or block-wise dynamic routing, LoRA-Switch introduces token-wise routing, switching LoRA adapters and weights for each token and merging them into the backbone for inference. An optimized CUDA kernel fuses merging operations for all LoRA adapters at once, achieving similar accuracy improvement as existing adapters while reducing decoding latency by more than 2.4 times. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps us make large language models (LLMs) better by adding special parts called dynamic adapters. These adapters are like super-chargers that give LLMs a boost, but they also make the model run slower. The authors looked at why this happens and found that it’s because of how the computer does some calculations. To fix this, they created a new way to make these adapters work more efficiently, called LoRA-Switch. It works by switching between different parts of the adapter for each little piece of text (called tokens) and then combining them to get the final answer. This makes the model run faster without sacrificing its accuracy. |
Keywords
» Artificial intelligence » Inference » Lora » Mixture of experts » Token
