Summary of Teaching Yourself: Graph Self-distillation on Neighborhood For Node Classification, by Lirong Wu et al.
Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification
by Lirong Wu, Jun Xia, Haitao Lin, Zhangyang Gao, Zicheng Liu, Guojiang Zhao, Stan Z. Li
First submitted to arxiv on: 5 Oct 2022
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Recent advances in Graph Neural Networks (GNNs) have led to impressive results in handling graph-related tasks. However, Multi-Layer Perceptrons (MLPs) remain the primary choice for practical industrial applications due to their faster inference times. The gap between academic and industrial success is attributed to GNN’s neighborhood-fetching latency, which makes it challenging to deploy in latency-sensitive applications. On the other hand, MLPs lack feature aggregation and infer slower than GNNs but outperform them in terms of performance. To bridge this gap, we propose a Graph Self-Distillation on Neighborhood (GSDN) framework, utilizing MLPs with structural information as prior to guide knowledge self-distillation between the neighborhood and target. This approach enjoys graph topology-awareness during training without data dependency at inference time. Experimental results show that GSDN improves vanilla MLP performance by 15.54% on average and outperforms state-of-the-art GNNs on six datasets, while inferring 75X-89X faster than existing GNNs and 16X-25X faster than other acceleration methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Researchers have made big progress in using computers to understand complex relationships between things. Graph Neural Networks (GNNs) are great at this, but they’re not always the best choice for real-world applications because they can be slow. On the other hand, Multi-Layer Perceptrons (MLPs) are faster but not as good at understanding relationships. To solve this problem, scientists have developed a new approach called Graph Self-Distillation on Neighborhood (GSDN). This method uses MLPs to learn from neighborhood information without needing to access all the data. The results show that GSDN can improve how well MLPs work and even outperform the best GNNs on some tasks, while also being much faster. |
Keywords
* Artificial intelligence * Distillation * Gnn * Inference
