Summary of Pecc: Problem Extraction and Coding Challenges, by Patrick Haller et al.
PECC: Problem Extraction and Coding Challenges
by Patrick Haller, Jonas Golde, Alan Akbik
First submitted to arxiv on: 29 Apr 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces PECC, a novel benchmark for evaluating large language models (LLMs) in code generation tasks. The benchmark is derived from Advent Of Code challenges and Project Euler, with 2396 problems that require LLMs to interpret narrative-embedded problems, extract requirements, and generate executable code. Unlike conventional benchmarks, PECC includes natural language prompting in chat-based evaluations, mirroring real-world instruction ambiguities. Results show varying model performance between narrative and neutral problems, with specific challenges in the Euler math-based subset. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper creates a new way to test big language models that can generate code. It’s called PECC, and it’s based on puzzles from Advent Of Code and Project Euler. The tests require the models to read a problem, figure out what needs to be done, and then write the correct code. This is different from how most models are tested now. The results show that some models do better with certain types of problems than others. |
Keywords
» Artificial intelligence » Prompting
