Summary of A Peek Into Token Bias: Large Language Models Are Not Yet Genuine Reasoners, by Bowen Jiang et al.
A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners
by Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J. Su, Camillo J. Taylor, Dan Roth
First submitted to arxiv on: 16 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed hypothesis-testing framework evaluates whether large language models (LLMs) rely on genuine reasoning abilities or token bias when solving logical reasoning tasks. The study introduces a novel approach that goes beyond accuracy evaluation, instead focusing on investigating LLMs’ token bias in conjunction fallacy and syllogistic problems. A list of hypotheses is developed to identify token biases, with null hypotheses assuming genuine reasoning capabilities. The results suggest that most LLMs struggle with logical reasoning, relying heavily on recognizing superficial patterns with strong token bias. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models (LLMs) are super smart computers that can understand and generate human-like text. But do they really think like humans? This study looks at how well LLMs can solve logic problems, like “All cats are animals” and “If all cats are animals, then all dogs are…?” Researchers created special test datasets with tricky questions to see if the LLMs were using real thinking or just memorizing patterns. They found that most LLMs were still struggling with these kinds of problems and mostly rely on recognizing patterns they’ve seen before. |
Keywords
» Artificial intelligence » Token
