Summary of Deep Neural Network Initialization with Sparsity Inducing Activations, by Ilan Price et al.
Deep Neural Network Initialization with Sparsity Inducing Activations
by Ilan Price, Nicholas Daultry Ball, Samuel C.H. Lam, Adam C. Jones, Jared Tanner
First submitted to arxiv on: 25 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates ways to improve the efficiency of deep neural networks by inducing sparse activations during training and inference. By analyzing the behavior of nonlinear activations that induce sparsity, the authors show that two popular activation functions, shifted ReLU and soft thresholding, can be unstable if not properly handled. They propose a solution by clipping the magnitude of these activations, which allows for high levels of sparsity (up to 85%) while retaining accuracy. The authors use the large width Gaussian process limit to analyze the behavior of these activations at random initialization. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Deep neural networks are getting bigger and more powerful, but they also get slower and harder to train. One way to make them faster is to make some parts of them “sleepy” so that they don’t have to do as much work. The paper looks at how this works for two special kinds of sleepy neurons called shifted ReLU and soft thresholding. It shows that these neurons can be tricky to train, but if you clip their strength just right, you can make them really sleepy (up to 85%) without losing any accuracy. |
Keywords
* Artificial intelligence * Inference * Relu
