Summary of Memory-efficient Vision Transformers: An Activation-aware Mixed-rank Compression Strategy, by Seyedarmin Azizi et al.
Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy
by Seyedarmin Azizi, Mahdi Nazemi, Massoud Pedram
First submitted to arxiv on: 8 Feb 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes an activation-aware model compression methodology to reduce the memory footprint of Vision Transformers (ViTs) for practical deployment on inference engines. The approach uses selective low-rank weight tensor approximations to compress ViT models, minimizing errors and achieving excellent results while avoiding local minima early in optimization. The method is demonstrated on DeiT-B, reducing its parameter count by 60% with less than 1% accuracy drop on ImageNet. Additionally, the compression technique can shrink large models to match smaller variants’ size while yielding up to 1.8% accuracy gain. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps solve a problem that stops us from using powerful Vision Transformers in everyday devices. These machines need lots of memory and computing power, which makes them hard to use on phones or other small computers. The researchers came up with a clever way to shrink the size of these models while keeping their performance almost the same. This means we might soon be able to use these powerful machines in more places. |
Keywords
» Artificial intelligence » Inference » Model compression » Optimization » Vit
