Summary of Random Relu Neural Networks As Non-gaussian Processes, by Rahul Parhi et al.
Random ReLU Neural Networks as Non-Gaussian Processes
by Rahul Parhi, Pakshal Bohra, Ayoub El Biari, Mehrsa Pourya, Michael Unser
First submitted to arxiv on: 16 May 2024
Categories
- Main: Machine Learning (stat.ML)
- Secondary: Machine Learning (cs.LG); Probability (math.PR)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper explores shallow neural networks with randomly initialized parameters and rectified linear unit activation functions, demonstrating that these networks are well-defined non-Gaussian processes. The study shows that these networks are solutions to stochastic differential equations driven by impulsive white noise and proves that they are isotropic and wide-sense self-similar with Hurst exponent 3/2. The researchers also derive a closed-form expression for the autocovariance function of these processes. Notably, this work considers a non-asymptotic viewpoint, where the number of neurons in each region is itself a random variable with a Poisson law. As the expected width tends to infinity, the processes can converge to both Gaussian and non-Gaussian distributions depending on the law of the weights. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at special kinds of neural networks that are not very deep. It shows that these networks are special types of mathematical processes called stochastic differential equations. The researchers found out some important things about how these networks behave, like how they’re related to other types of random processes. They also came up with a simple formula for understanding how these networks work together over time. What’s unique about this study is that it looks at the number of neurons in each part of the network as itself being random and changing. This means we can learn more about what happens when we have many layers or connections in our neural networks. |
