Summary of The Need For Speed: Pruning Transformers with One Recipe, by Samir Khaki and Konstantinos N. Plataniotis
The Need for Speed: Pruning Transformers with One Recipe
by Samir Khaki, Konstantinos N. Plataniotis
First submitted to arxiv on: 26 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The One-shot Pruning Technique for Interchangeable Networks (OPTIN) framework is a tool to increase the efficiency of pre-trained transformer architectures without requiring re-training. The OPTIN framework leverages intermediate feature distillation, capturing long-range dependencies of model parameters called trajectories, to produce state-of-the-art results on natural language, image classification, transfer learning, and semantic segmentation tasks without re-training. Given a FLOP constraint, the OPTIN framework compresses the network while maintaining competitive accuracy performance and improved throughput. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The OPTIN framework is a new way to make transformer architectures more efficient without needing to retrain them. It uses something called feature distillation to capture how model parameters work together, which allows it to produce good results on different tasks like language and image recognition, transfer learning, and semantic segmentation. The framework can reduce the number of calculations needed while keeping accuracy high. |
Keywords
* Artificial intelligence * Distillation * Image classification * One shot * Pruning * Semantic segmentation * Transfer learning * Transformer
