Summary of Apt: Adaptive Pruning and Tuning Pretrained Language Models For Efficient Training and Inference, by Bowen Zhao et al.
APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference
by Bowen Zhao, Hannaneh Hajishirzi, Qingqing Cao
First submitted to arxiv on: 22 Jan 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach to large Language Model (LM) fine-tuning and inference is proposed, which adaptively prunes and tunes parameters for efficiency. The method, called APT, dynamically adds important tuning parameters at the early stages of fine-tuning to achieve fast and accurate convergence, while discarding less important ones to reduce computational cost and memory usage. Compared to baseline methods, APT achieves up to 98% task performance with RoBERTa and T5 models when pruning 40% of their original parameters, and maintains 86.4% performance with LLaMA models when retaining 70% of their original parameters. Additionally, APT reduces fine-tuning time by up to 8x and memory usage by up to 70%. This technique has the potential to significantly improve the efficiency of large LM-based applications. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large Language Models (LMs) are powerful tools for natural language processing tasks, but they require a lot of computing power and memory. A new way to make LMs work better is called APT, which adds or removes parts of the model as needed to balance performance and efficiency. When we tested APT with popular models like RoBERTa, T5, and LLaMA, it did really well, keeping most of their original performance while using much less memory and computation time. |
Keywords
* Artificial intelligence * Fine tuning * Inference * Large language model * Llama * Natural language processing * Pruning * T5
