Summary of Wilt: a Multi-turn, Memorization-robust Inductive Logic Benchmark For Llms, by Eryk Banatt et al.
WILT: A Multi-Turn, Memorization-Robust Inductive Logic Benchmark for LLMs
by Eryk Banatt, Jonathan Cheng, Skanda Vaidyanath, Tiffany Hwu
First submitted to arxiv on: 14 Oct 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Large language models (LLMs) have achieved impressive results across various domains, but they still struggle with reasoning tasks that require evidence gathering and logical conclusions. This limitation affects chat user interfaces that rely on multi-turn interactions for effective collaboration. To address this, we introduce the Wason Inductive Logic Test (WILT), a benchmark designed to resist memorization. WILT is inspired by the Wason 2-4-6 task, where participants must infer a boolean function involving three variables. In WILT, each test starts from a clean slate, with only initial instructions provided, preventing models from relying on pre-learned responses. Our findings reveal that LLMs struggle with this task, exhibiting distinct strengths and weaknesses. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are very smart at doing lots of things, but they’re still not great at figuring out tricky problems that require gathering clues and making logical conclusions. This makes it hard for chatbots to help customers solve problems over several turns. To fix this, we created a test called WILT that’s harder for machines to cheat on. It’s like a puzzle where the model has to come up with ideas to test its guesses and figure out the correct answer. |
