Summary of Lossless Model Compression Via Joint Low-rank Factorization Optimization, by Boyang Zhang et al.
Lossless Model Compression via Joint Low-Rank Factorization Optimization
by Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangmin Liu, Jiake Tian
First submitted to arxiv on: 9 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computational Complexity (cs.CC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a novel joint optimization strategy for low-rank factorization of weight matrices in deep neural networks, addressing the issue of performance discrepancy between model compression and optimization objectives. By analyzing the relationship between low-rank factorization and model optimization objectives, the authors establish a precise perturbation range for matrix factorization errors on model performance. This is reformulated as a numerical rank deficiency problem with inequality constraints, leading to two optimization algorithms: lossless optimization that maximizes model accuracy while ensuring compression, and compact optimization that minimizes model size while preserving performance. The proposed methods demonstrate robust efficacy across various vision and language tasks, achieving lossless results with reduced model sizes. For example, the compressed ResNext50 model outperforms the original with a 70% reduction in size. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper solves a problem in artificial intelligence by making neural networks smaller without losing their ability to work well. Currently, when we make these networks smaller, they can’t be as good at doing their job. The authors found a way to fix this by combining two important tasks: making the network smaller and making it better at its job. They created new methods that don’t require us to fine-tune anything, so we can easily use them on many different types of networks. These new methods work well across various tasks such as image recognition and language processing. |
Keywords
» Artificial intelligence » Model compression » Optimization
