Summary of Automatic Bug Detection in Llm-powered Text-based Games Using Llms, by Claire Jin et al.
Automatic Bug Detection in LLM-Powered Text-Based Games Using LLMs
by Claire Jin, Sudha Rao, Xiangyu Peng, Portia Botchway, Jessica Quaye, Chris Brockett, Bill Dolan
First submitted to arxiv on: 6 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Software Engineering (cs.SE)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel method for detecting bugs in large language model (LLM) powered interactive games is proposed. The approach uses player game logs to automatically identify issues such as hallucinations, forgetfulness, or misinterpretations of prompts that can cause logical inconsistencies and unexpected deviations from intended designs. This method surpasses unstructured LLM-powered bug-catching methods and fills the gap in automated detection of logical and design flaws. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are changing game design by enabling dynamic stories and interactions between players and non-player characters. However, these models sometimes make mistakes that can cause problems with the game’s storyline or character behavior. The paper proposes a new way to automatically find these errors using player logs from games like DejaBoom!. This approach is better than just relying on the language model to catch its own mistakes. |
Keywords
» Artificial intelligence » Language model » Large language model
