Summary of An Analytic Solution to Covariance Propagation in Neural Networks, by Oren Wright et al.
An Analytic Solution to Covariance Propagation in Neural Networks
by Oren Wright, Yorie Nakahira, José M. F. Moura
First submitted to arxiv on: 24 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel method for uncertainty quantification of neural networks, which is crucial for assessing their reliability and robustness. The authors introduce a sample-free moment propagation technique that accurately characterizes the input-output distributions of neural networks without relying on costly or inaccurate sampling methods. The key innovation lies in an analytic solution for the covariance of random variables passed through nonlinear activation functions, such as Heaviside, ReLU, and GELU. This technique has wide applicability and is demonstrated to be effective in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps us understand how to measure the reliability of artificial intelligence systems that use deep learning. Right now, it’s hard to do this without using expensive or inaccurate methods. The authors come up with a new way to do this without needing samples. This is made possible by solving a tricky math problem for random variables that go through certain types of computer code. This technique has lots of possibilities and is shown to work well in experiments. |
Keywords
* Artificial intelligence * Deep learning * Relu
