Summary of Beaver: An Enterprise Benchmark For Text-to-sql, by Peter Baile Chen et al.
BEAVER: An Enterprise Benchmark for Text-to-SQL
by Peter Baile Chen, Fabian Wenz, Yi Zhang, Devin Yang, Justin Choi, Nesime Tatbul, Michael Cafarella, Çağatay Demiralp, Michael Stonebraker
First submitted to arxiv on: 3 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Databases (cs.DB)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces a new dataset called BEAVER, which aims to bridge the gap between existing text-to-SQL benchmarks and real-world enterprise settings. The authors argue that existing benchmarks are constructed from web tables with human-generated question-SQL pairs, leading to strong results for Large Language Models (LLMs) on these tasks. However, this may not translate well to enterprises, where table structures and contents differ substantially. To contend with this issue, the authors collect natural language queries and their correct SQL statements from actual query logs in private enterprise data warehouses. They then benchmark off-the-shelf LLMs on this dataset and identify three main reasons for poor performance: complex schemas, business-oriented questions requiring joins and aggregations, and limited access to private enterprise data. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper makes a new dataset called BEAVER that helps computers understand what people mean when they ask questions about company databases. Right now, most computer models are good at understanding simple questions from the internet. But real companies have different kinds of data and more complex questions. This makes it hard for computers to help with tasks like finding information or solving problems in these companies’ databases. The authors made this new dataset by collecting actual questions people asked about company databases and the correct answers. They tested popular computer models on this new dataset and found that they did poorly because they’re not used to dealing with real-world data and complex questions. |
