Summary of Preserving Deep Representations in One-shot Pruning: a Hessian-free Second-order Optimization Framework, by Ryan Lucas et al.
Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework
by Ryan Lucas, Rahul Mazumder
First submitted to arxiv on: 27 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel post-training pruning framework called SNOWS is introduced for reducing the computational cost of vision network inference without retraining. Unlike current leading one-shot pruning methods that minimize layer-wise least squares reconstruction error, SNOWS optimizes a more global reconstruction objective that accounts for nonlinear activations deep in the network. This leads to a more challenging optimization problem, which can be efficiently solved using a specialized second-order optimization framework. A key innovation of SNOWS is its ability to use Hessian-free optimization to compute exact Newton descent steps without requiring the full Hessian matrix. Additionally, SNOWS can be applied on top of any sparse mask derived from prior methods, readjusting their weights to exploit nonlinearities in deep feature representations. The framework achieves state-of-the-art results on various one-shot pruning benchmarks, including residual networks and Vision Transformers (ViT/B-16 and ViT/L-16). |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary SNOWS is a new way to make computer vision models work faster without needing to retrain them from scratch. Current methods try to fix the model by minimizing how much it changes, but this doesn’t account for deeper parts of the network. SNOWS fixes this problem by looking at the whole network and adjusting its weights to use more complex features. This makes the model work better and faster. |
Keywords
» Artificial intelligence » Inference » Mask » One shot » Optimization » Pruning » Vit
