Summary of Evaluating Large Language Models with Grid-based Game Competitions: An Extensible Llm Benchmark and Leaderboard, by Oguzhan Topsakal et al.
Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboard
by Oguzhan Topsakal, Colby Jacob Edell, Jackson Bailey Harper
First submitted to arxiv on: 10 Jul 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces a novel benchmark for large language models (LLMs) through grid-based games such as Tic-Tac-Toe, Connect Four, and Gomoku. The open-source code allows LLMs to compete, generating detailed data files for leaderboard rankings and analysis. Leading LLMs from Anthropic, Google, OpenAI, and Meta are compared, simulating 2,310 matches across three game types and prompt types. The results reveal significant variations in performance, with analysis covering win rates, missed opportunities, and invalid moves. This study assesses LLM capabilities, rule comprehension, and strategic thinking, laying groundwork for future exploration into decision-making scenarios and AGI. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study looks at how well big language models can play games they weren’t trained for. It uses open-source code to let the models compete in games like Tic-Tac-Toe and Connect Four. The results show that different models do much better or worse depending on the game and how it’s set up. This helps us understand how good these models are at following rules and making smart choices. |
Keywords
* Artificial intelligence * Prompt
