Summary of Regularized Kl-divergence For Well-defined Function-space Variational Inference in Bayesian Neural Networks, by Tristan Cinquin and Robert Bamler
Regularized KL-Divergence for Well-Defined Function-Space Variational Inference in Bayesian neural networks
by Tristan Cinquin, Robert Bamler
First submitted to arxiv on: 6 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Bayesian neural networks (BNNs) aim to integrate the predictive capabilities of neural networks with principled uncertainty modeling, essential for safety-critical applications and decision-making. However, posterior uncertainty estimates rely on prior choices, making it challenging to find informative priors in weight-space. Variational inference (VI) methods address this issue by posing priors directly on the function generated by the BNN rather than weights. Our study resolves a fundamental issue with VI approaches in function-space inference for BNNs with Gaussian process (GP) priors, as highlighted by Burt et al. (2020). We introduce generalized VI with regularized KL divergence, providing the first well-defined variational objective for function-space inference in BNNs. Our method demonstrates competitive uncertainty estimates on synthetic and small real-world datasets for regression, classification, and out-of-distribution detection tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine a special kind of neural network that can predict things while also telling you how sure it is about its predictions. This could be super helpful in situations where decisions are critical or uncertain. But making this work requires finding the right “prior” information, which has been tricky. Some researchers have tried to solve this problem by looking at the function that the neural network produces rather than the individual weights. Our team discovered a major issue with this approach and came up with a solution using something called generalized variational inference. We tested our method on some fake data and a small real-world dataset, and it performed well for tasks like predicting numbers or classifying things. |
Keywords
* Artificial intelligence * Classification * Inference * Neural network * Regression
