Summary of Laco: Large Language Model Pruning Via Layer Collapse, by Yifei Yang et al.
LaCo: Large Language Model Pruning via Layer Collapse
by Yifei Yang, Zouying Cao, Hai Zhao
First submitted to arxiv on: 17 Feb 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Layer Collapse (LaCo) method is a concise layer-wise structured pruner that reduces the size of large language models (LLMs) based on transformers while preserving their structure. The LaCo method collapses rear model layers into prior layers, enabling rapid size reduction and outperforming existing state-of-the-art structured pruning methods. Comprehensive experiments show an average task performance of over 80% at pruning ratios of 25-30%. Post-training experiments confirm that LaCo effectively inherits the parameters of the original model. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are getting bigger, which makes them expensive to train and use. Researchers have tried different ways to make these models smaller without losing their abilities. However, these methods have limitations, such as needing a lot of training data or changing how the model works inside. A new method called LaCo tries to solve this problem by collapsing some parts of the model into others. This helps reduce the size of the model while keeping its original structure. The results show that LaCo is better than other methods at preserving the model’s abilities when it gets smaller. |
Keywords
» Artificial intelligence » Pruning
