Summary of Cogbench: a Large Language Model Walks Into a Psychology Lab, by Julian Coda-forno et al.
CogBench: a large language model walks into a psychology lab
by Julian Coda-Forno, Marcel Binz, Jane X. Wang, Eric Schulz
First submitted to arxiv on: 28 Feb 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces CogBench, a novel benchmark for evaluating large language models (LLMs) based on ten behavioral metrics derived from seven cognitive psychology experiments. Unlike traditional benchmarks that focus solely on performance metrics, CogBench provides a comprehensive toolkit for phenotyping LLMs’ behavior. The authors apply CogBench to 35 LLMs and analyze the data using statistical multilevel modeling techniques. Their study highlights the importance of model size and reinforcement learning from human feedback (RLHF) in improving performance and aligning with human behavior. Interestingly, they find that open-source models are less risk-prone than proprietary models and that fine-tuning on code does not necessarily enhance LLMs’ behavior. The authors also explore the effects of prompt-engineering techniques, discovering that chain-of-thought prompting improves probabilistic reasoning and take-a-step-back prompting fosters model-based behaviors. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about a new way to test how well language models work. Right now, people mostly test them by seeing how well they can answer questions or do tasks. But this approach doesn’t tell us everything we want to know. The authors of the paper created something called CogBench, which looks at ten different ways that language models behave. They tested 35 different models using this new approach and found some interesting things. For example, bigger models are better at doing certain tasks, and when people help train them by giving feedback, they get even better. The authors also found that open-source models (which anyone can use) are safer to use than proprietary models (which only big companies have access to). They even experimented with different ways of asking the models questions and found that some methods work better than others. |
Keywords
* Artificial intelligence * Fine tuning * Prompt * Prompting * Reinforcement learning from human feedback * Rlhf
