Summary of Residual Connections and Normalization Can Provably Prevent Oversmoothing in Gnns, by Michael Scholkemper et al.
Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs
by Michael Scholkemper, Xinyi Wu, Ali Jadbabaie, Michael T. Schaub
First submitted to arxiv on: 5 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper provides a theoretical understanding of how residual connections and normalization layers alleviate the oversmoothing problem in graph neural networks (GNNs). The authors show that residual connections prevent the signal from becoming too smooth by incorporating initial features at each layer, determining the subspace of possible node representations. Batch normalization prevents the collapse of the output embedding space to a one-dimensional subspace through individual rescaling of feature matrix columns. Additionally, the centering step of traditional normalization layers can distort the original graph signal. The authors introduce GraphNormv2, a novel normalization layer that learns to center without distorting the signal. Experimental results confirm the effectiveness of this method. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps us understand how GNNs work better with residual connections and normalization layers. It shows that these techniques can prevent important information from getting lost in calculations. The authors also introduce a new way to normalize data, called GraphNormv2, which is designed not to mess up the original signal. |
Keywords
» Artificial intelligence » Batch normalization » Embedding space
