Summary of How Far Are We on the Decision-making Of Llms? Evaluating Llms’ Gaming Ability in Multi-agent Environments, by Jen-tse Huang et al.
How Far Are We on the Decision-Making of LLMs? Evaluating LLMs’ Gaming Ability in Multi-Agent Environments
by Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, Michael R. Lyu
First submitted to arxiv on: 18 Mar 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A new framework for evaluating Large Language Models’ decision-making abilities in complex, multi-agent environments is proposed. The GAMA()-Bench framework includes eight game theory scenarios and a dynamic scoring scheme to assess performance, robustness, generalizability, and strategies for improvement. The authors evaluate 13 LLMs from six model families, including GPT-3.5, Gemini, LLaMA-3.1, Mixtral, Qwen-2, and find that Gemini-1.5-Pro outperforms others with a score of 69.8 out of 100. The results suggest that GPT-3.5 demonstrates strong robustness but limited generalizability, which can be improved using methods like Chain-of-Thought. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large Language Models are being tested to see how well they make decisions. This is important because real-life decision-making involves many factors and people working together. The current way of testing these models only looks at two-player games, where one model plays against another. But this doesn’t give us a complete picture. To fix this, researchers have created GAMA()-Bench, a new framework that tests the models in different game scenarios and gives them scores based on how well they do. The results show that some models are better at making decisions than others. |
Keywords
» Artificial intelligence » Gemini » Gpt » Llama
