Summary of Pruner-zero: Evolving Symbolic Pruning Metric From Scratch For Large Language Models, by Peijie Dong and Lujun Li and Zhenheng Tang and Xiang Liu and Xinglin Pan and Qiang Wang and Xiaowen Chu
Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models
by Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, Xiaowen Chu
First submitted to arxiv on: 5 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL); Neural and Evolutionary Computing (cs.NE)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a novel approach to efficiently identify superior pruning metrics for Large Language Models (LLMs) without retraining. The authors develop an automatic framework using genetic programming, called Pruner-Zero, which searches for symbolic pruning metrics and generates them automatically. The framework includes an elaborate search space that encompasses existing pruning metrics, allowing it to discover potential new ones. To increase diversity, the authors propose an opposing operation simplification strategy. Extensive experiments on LLaMA and LLaMA-2 demonstrate that Pruner-Zero outperforms state-of-the-art post-training pruning methods in language modeling and zero-shot tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Pruning large language models is challenging because it requires retraining, which is expensive and computationally demanding. Researchers have developed new metrics to prune the model without retraining, but these require human experts and trial-and-error. This paper presents a way to automatically find the best pruning method using genetic programming. The authors test their approach on two large language models and show that it works better than other methods. |
Keywords
» Artificial intelligence » Llama » Pruning » Zero shot
