Summary of Learn to Be Efficient: Build Structured Sparsity in Large Language Models, by Haizhong Zheng et al.
Learn To be Efficient: Build Structured Sparsity in Large Language Models
by Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Z. Morley Mao, Beidi Chen, Fan Lai, Atul Prakash
First submitted to arxiv on: 9 Feb 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces a novel training algorithm called Learn-To-be-Efficient (LTE) to train large language models (LLMs) to learn structured activation sparsity, reducing inference costs. LTE is designed for LLMs like LLaMA using non-ReLU activations and outperforms state-of-the-art baselines on language understanding, generation, and instruction tuning tasks. The algorithm achieves a better trade-off between sparsity and performance by learning to activate fewer neurons. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models (LLMs) have become very good at processing text, but they use up a lot of computer power to do it. Researchers are looking for ways to make them more efficient without losing their ability to understand and generate human-like text. One idea is to only use parts of the model’s “brain” when needed. This paper introduces a new way to train these models to be more efficient, which can also reduce the time it takes to process text. |
Keywords
* Artificial intelligence * Inference * Instruction tuning * Language understanding * Llama * Relu
