Summary of Not All Llm Reasoners Are Created Equal, by Arian Hosseini et al.
Not All LLM Reasoners Are Created Equal
by Arian Hosseini, Alessandro Sordoni, Daniel Toyama, Aaron Courville, Rishabh Agarwal
First submitted to arxiv on: 2 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary We investigate the ability of Large Language Models (LLMs) to solve grade-school math problems that require sequential reasoning. To assess this capability, we design pairs of math word problems where the answer to the second problem relies on correctly solving the first problem. Our results show a significant gap in LLM performance when solving compositional pairs versus individual problems, with smaller models exhibiting more pronounced gaps. Interestingly, instruction-tuning and code generation have varying effects depending on model size, while fine-tuning on grade-school math can lead to task overfitting. Our analysis suggests that large reasoning gaps are not due to test-set leakage, but rather to distraction from additional context and poor second-hop reasoning. Overall, our findings highlight systematic differences in LLMs’ reasoning abilities, which may not be reflected in their performance on standard benchmarks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary We studied how well Large Language Models (LLMs) can solve math problems that require thinking ahead. We created pairs of math questions where the answer to the second question depends on solving the first one correctly. Our results show that most LLMs struggle with these types of problems, especially smaller ones. We also found that different approaches to training these models have varying effects on their ability to solve these types of math problems. Overall, our study highlights the limitations of current AI systems in understanding and solving complex math problems. |
Keywords
» Artificial intelligence » Fine tuning » Instruction tuning » Overfitting
