Summary of Erbench: An Entity-relationship Based Automatically Verifiable Hallucination Benchmark For Large Language Models, by Jio Oh et al.
ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models
by Jio Oh, Soyeon Kim, Junseok Seo, Jindong Wang, Ruochen Xu, Xing Xie, Steven Euijong Whang
First submitted to arxiv on: 8 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach to evaluating large language models (LLMs) using relational databases based on the entity-relationship (ER) model. Existing benchmarks lack the ability to evaluate LLMs with arbitrary complexity, but ERBench addresses this issue by utilizing integrity constraints from these databases. Specifically, functional dependencies pinpoint critical keywords for answering complex questions, while foreign key constraints enable multi-hop questions to debug intermediate answers. The proposed benchmark supports continuous evaluation as databases change and allows for multimodal questions and various prompt engineering techniques. In experiments, the authors construct LLM benchmarks using databases of multiple domains and compare contemporary LLMs, demonstrating ERBench’s effectiveness in evaluating LLMs by verifying both answer correctness and rationales. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are very smart computers that can understand and generate human-like text. But it’s hard to know if they’re really good or not. The authors of this paper think that using special kinds of databases called relational databases could be a way to test these models. These databases contain structured information that can be used to create complex questions for the models, which is important because real-world problems often require understanding many different pieces of information. The proposed system, ERBench, allows for creating benchmarks (tests) using any database, and it’s flexible enough to accommodate changes in the databases as well as different types of questions. The authors demonstrate how ERBench can be used to evaluate various language models by not only checking if they give the correct answer but also looking at their thought process. |
Keywords
* Artificial intelligence * Prompt
