Summary of Gamebench: Evaluating Strategic Reasoning Abilities Of Llm Agents, by Anthony Costarelli et al.
GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
by Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, Arjun Yadav
First submitted to arxiv on: 7 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a comprehensive framework for evaluating the strategic reasoning abilities of large language model (LLM) agents across various types of games. The proposed benchmark, GameBench, consists of 9 different game environments that cover at least one axis of key reasoning skills identified in strategy games. The authors use GPT-3 and GPT-4 models to demonstrate the effectiveness of two scaffolding frameworks: Chain-of-Thought (CoT) prompting and Reasoning Via Planning (RAP). Results show that none of the tested models match human performance, with CoT and RAP improving scores but not reaching comparable levels. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models have been shown to perform well on many natural language understanding tasks. But there hasn’t been a way to compare how good they are at making decisions in different types of games. This paper introduces a new benchmark that tests the ability of these models to make strategic decisions. The authors used two kinds of scaffolding to help the models do better: one helps them think step-by-step, and the other helps them plan ahead. While the models did get better with the help, they still didn’t perform as well as humans. |
Keywords
» Artificial intelligence » Gpt » Language understanding » Large language model » Prompting
