Summary of A Survey on Transformer Compression, by Yehui Tang et al.
A Survey on Transformer Compression
by Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, Dacheng Tao
First submitted to arxiv on: 5 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, researchers investigate the role of Transformer models in natural language processing (NLP) and computer vision (CV), particularly with regards to constructing large language models (LLMs) and large vision models (LVMs). The authors highlight the importance of model compression methods, which reduce the memory and computational cost of Transformer-based models, making them more practical for implementation on devices. The paper surveys recent compression techniques, focusing on their application to Transformer-based models, and categorizes these methods into pruning, quantization, knowledge distillation, and efficient architecture design. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Transformer is a powerful tool in NLP and CV that helps build big language and vision models. To make these models work on regular devices, we need to shrink them down while keeping their performance good. The researchers looked at different ways to do this, like cutting out parts of the model, using simpler numbers, teaching smaller models, or designing new architectures. They also explored how these methods can be used for both language and vision tasks. |
Keywords
* Artificial intelligence * Knowledge distillation * Model compression * Natural language processing * Nlp * Pruning * Quantization * Transformer
