Summary of Glbench: a Comprehensive Benchmark For Graph with Large Language Models, by Yuhan Li et al.
GLBench: A Comprehensive Benchmark for Graph with Large Language Models
by Yuhan Li, Peisong Wang, Xiao Zhu, Aochuan Chen, Haiyun Jiang, Deng Cai, Victor Wai Kin Chan, Jia Li
First submitted to arxiv on: 10 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces GLBench, a comprehensive benchmark for evaluating GraphLLM methods in supervised and zero-shot scenarios. This is the first benchmark to provide consistent experimental protocols for fair evaluation of different categories of GraphLLM methods, including traditional baselines like graph neural networks. The authors conduct extensive experiments on real-world datasets with consistent data processing and splitting strategies. Key findings include: GraphLLM methods outperform traditional baselines in supervised settings, with LLM-as-enhancers showing robust performance; however, using LLMs as predictors is less effective and may lead to uncontrollable output issues. The study also highlights the importance of both structures and semantics for effective zero-shot transfer, with a simple baseline able to outperform some models tailored for zero-shot scenarios. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A new way to understand graphs has been developed using large language models (LLMs). However, there was no standard way to test how well these methods work. To fix this problem, the authors created GLBench, a tool that helps compare different LLM methods. They tested many of these methods on real-world datasets and found some surprising results. For example, they found that some LLM methods are really good at predicting what will happen in a graph, but others can get stuck in loops. The study also showed that it’s important to consider both the structure and meaning of the graph when trying to make predictions. |
Keywords
* Artificial intelligence * Semantics * Supervised * Zero shot
