Summary of Slope: Double-pruned Sparse Plus Lazy Low-rank Adapter Pretraining Of Llms, by Mohammad Mozaffari et al.
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs
by Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang, Maryam Mehri Dehnavi
First submitted to arxiv on: 25 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes SLoPe, a novel method for improving the accuracy of sparse large language models (LLMs) while accelerating pretraining and inference. The approach addresses the trade-off between model sparsity and accuracy by adding low-rank adapters in the final stages of pretraining, without significant overheads. SLoPe also employs a double-pruned backward pass formulation to accelerate sparse backward passes. Experimental results show that SLoPe achieves up to 1.25x and 1.54x speedup in training and inference, respectively, for models with billions of parameters, while reducing memory usage by up to 0.63x and 0.61x. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary SLoPe is a new way to make large language models faster and more efficient. It helps fix the problem of these models being too big and using too much memory. SLoPe does this by adding special adapters that help the model learn better, without making it use more energy or take longer to train. The idea works so well that it can speed up training by 25% and make models run 54% faster when they’re used to generate text. It also makes these big models take up less space on computers. |
Keywords
» Artificial intelligence » Inference » Pretraining
