Summary of Sgformer: Simplifying and Empowering Transformers For Large-graph Representations, by Qitian Wu et al.
SGFormer: Simplifying and Empowering Transformers for Large-Graph Representations
by Qitian Wu, Wentao Zhao, Chenxiao Yang, Hengrui Zhang, Fan Nie, Haitian Jiang, Yatao Bian, Junchi Yan
First submitted to arxiv on: 19 Jun 2023
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Social and Information Networks (cs.SI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents a novel approach to learning representations on large-sized graphs by simplifying the architecture of transformers, which are typically used in language and vision tasks. The authors demonstrate that a single-layer attention mechanism can achieve competitive performance on node property prediction benchmarks, even on massive datasets with billions of nodes. This simplicity is achieved by eliminating positional encodings, feature/graph pre-processing, and augmented loss functions. The proposed scheme, Simplified Graph Transformers (SGFormer), requires only one layer of attention to propagate information among arbitrary nodes efficiently. SGFormer is shown to scale well to web-scale graphs and achieves up to 141x inference acceleration over state-of-the-art transformers on medium-sized graphs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper shows that we don’t need complicated models to learn representations on big graphs. By using a simple attention mechanism, we can get good results even with massive datasets. This is important because it means we can use these methods for really big graphs, like the ones used in social media or the internet. |
Keywords
* Artificial intelligence * Attention * Inference
