Summary of Domaineval: An Auto-constructed Benchmark For Multi-domain Code Generation, by Qiming Zhu et al.
DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation
by Qiming Zhu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, Shing-Chi Cheung
First submitted to arxiv on: 23 Aug 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Software Engineering (cs.SE)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a new benchmark, DOMAINEVAL, to evaluate the capabilities of Large Language Models (LLMs) on domain-specific coding tasks. Current benchmarks primarily focus on common coding tasks, leaving these tasks unexplored. The proposed pipeline works in an automated manner, enabling construction from code repositories into formatted subjects under study. The authors evaluate 12 representative LLMs against DOMAINEVAL and find that LLMs are generally good at computation tasks but fall short on cryptography and system coding tasks. They also observe that generating more samples can increase overall performance while domain bias may even increase. The contributions include a code generation benchmark dataset, a fully automated pipeline, and an identification of the limitations of LLMs in code generation tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper creates a new way to test how well Large Language Models (LLMs) can do coding tasks that are specific to certain areas. Right now, most tests just focus on simple coding problems, but this one tries to cover more advanced and specialized topics. The authors use a special system to make the testing happen automatically and then tested 12 different LLMs using their new benchmark. They found that some LLMs are really good at doing certain types of coding tasks, like math and calculations, but struggle with others, like making sure computer systems are secure. This helps us understand what LLMs can do well and where they need improvement. |
