Summary of Super: Evaluating Agents on Setting Up and Executing Tasks From Research Repositories, by Ben Bogin et al.
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
by Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot
First submitted to arxiv on: 11 Sep 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL); Software Engineering (cs.SE)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces SUPER, a benchmark designed to evaluate the capability of Large Language Models (LLMs) in setting up and executing tasks from research repositories. The authors aim to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. The benchmark consists of three problem sets: 45 end-to-end problems, 152 sub-problems, and 602 automatically generated problems. The authors propose various evaluation measures to assess task success and progress. They show that state-of-the-art approaches struggle to solve these problems, with the best model (GPT-4o) solving only 16.3% of the end-to-end set and 46.1% of the scenarios. This highlights the challenge of this task and suggests that SUPER can serve as a valuable resource for the community. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about creating a way to test how well Large Language Models (LLMs) can help researchers by reproducing results from research papers. Right now, LLMs are good at writing code, but they can’t do this task on their own. The authors created a benchmark called SUPER that includes many problems for LLMs to try and solve. They want to see how well these models can understand and follow instructions from research papers. So far, the best model can only solve about 16% of the problems correctly. This shows just how hard this task is and why it’s important to have a benchmark like SUPER. |
Keywords
» Artificial intelligence » Gpt » Machine learning » Natural language processing » Nlp
