Summary of Disp-llm: Dimension-independent Structural Pruning For Large Language Models, by Shangqian Gao and Chi-heng Lin and Ting Hua and Tang Zheng and Yilin Shen and Hongxia Jin and Yen-chang Hsu
DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models
by Shangqian Gao, Chi-Heng Lin, Ting Hua, Tang Zheng, Yilin Shen, Hongxia Jin, Yen-Chang Hsu
First submitted to arxiv on: 15 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach to large language model (LLM) pruning, addressing the challenge of deploying LLMs on resource-limited devices. The method, called dimension-independent structural pruning, relaxes constraints imposed by traditional methods and eliminates dependence along the embedding dimension. This allows for varying widths in different blocks, increasing flexibility. The authors evaluate their approach on various LLMs, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2, demonstrating improved performance compared to state-of-the-art methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making big language models smaller and more efficient. Right now, these models are very powerful but take up a lot of space on computers and use a lot of energy. The researchers developed a new way to make the models smaller without sacrificing their abilities. This approach lets different parts of the model use different information, which makes it more flexible. They tested this method with several big language models and found that it worked better than other approaches. |
Keywords
» Artificial intelligence » Embedding » Large language model » Llama » Pruning
