Summary of Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors For Bayesian Neural Networks, by Yoav Gelberg et al.
Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networks
by Yoav Gelberg, Tycho F.A. van der Ouderaa, Mark van der Wilk, Yarin Gal
First submitted to arxiv on: 10 Aug 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel investigation into the impact of weight space symmetries on variational inference (VI) techniques is presented, focusing on Bayesian neural networks (BNNs). The authors demonstrate that these symmetries lead to biased approximate posteriors, degrading predictive performance and fit if not explicitly addressed. To mitigate this issue, a symmetrization mechanism for constructing permutation-invariant variational posteriors is proposed, leveraging the symmetric structure of the posterior. This approach is shown to result in improved predictions and a higher ELBO objective. The authors train their model using a modified KL regularization term and demonstrate experimentally that it outperforms traditional VI methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A group of scientists studied how a special type of neural network works. They found that these networks can have many different possible answers, which makes it hard for computers to figure out the correct one. The researchers developed a new way to make the computer’s guesses better by taking into account the symmetries in the neural network. This approach was tested and showed that it produces more accurate results than traditional methods. |
Keywords
» Artificial intelligence » Inference » Neural network » Regularization
