Summary of Shaving Weights with Occam’s Razor: Bayesian Sparsification For Neural Networks Using the Marginal Likelihood, by Rayen Dhahri et al.
Shaving Weights with Occam’s Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood
by Rayen Dhahri, Alexander Immer, Betrand Charpentier, Stephan Günnemann, Vincent Fortuin
First submitted to arxiv on: 25 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a novel approach called SpaM (Sparsifiability via the Marginal likelihood) to improve the sparsification of neural networks, which is crucial for deploying AI models on consumer hardware. The framework combines Bayesian marginal likelihood with sparsity-inducing priors to create an automatic Occam’s razor that selects the most sparsifiable model while maintaining data explanation quality. Additionally, it utilizes a pre-computed posterior Hessian approximation from the Laplace approximation as a cheap pruning criterion, outperforming many existing methods. The paper demonstrates the effectiveness of SpaM across various neural network architectures and datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A team of researchers developed a new way to make artificial intelligence (AI) models smaller and more efficient. This is important because big AI models can take up too much space on computers and phones. They created an approach called SpaM that helps keep the information in AI models while making them smaller. The method uses special math calculations to figure out which parts of the model are most important and can be safely removed. The team tested their approach with different types of AI models and data, and it worked well even when they made the models much smaller. |
Keywords
* Artificial intelligence * Likelihood * Neural network * Pruning
