Summary of Peeking Behind the Curtains Of Residual Learning, by Tunhou Zhang et al.
Peeking Behind the Curtains of Residual Learning
by Tunhou Zhang, Feng Yan, Hai Li, Yiran Chen
First submitted to arxiv on: 13 Feb 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the fundamental principles behind residual learning in deep neural networks. It identifies a phenomenon called “dissipating inputs,” where plain layers compromise the input data through non-linearities, making it challenging for the network to learn feature representations. The authors theoretically demonstrate how plain networks degenerate the input to random noise and propose the “Plain Neural Net Hypothesis” (PNNH) as a solution. PNNH emphasizes the importance of residual connections in maintaining a lower bound of surviving neurons. The paper also presents CNN architectures and Transformers that implement PNNH, achieving on-par accuracy with ResNets and vision Transformers while improving training throughput and parameter efficiency. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at why some neural networks can learn things more easily than others. It finds out that when you have many layers in a network, the information going into each layer gets changed and becomes less useful. This makes it hard for the network to learn what it’s supposed to learn. The authors suggest a way to fix this problem by adding special connections between some of the layers. They test their idea on some common image recognition tasks and find that it works just as well as other methods, but is faster and uses fewer computer resources. |
Keywords
* Artificial intelligence * Cnn
