Summary of What Makes In-context Learning Effective For Mathematical Reasoning: a Theoretical Analysis, by Jiayu Liu et al.
What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis
by Jiayu Liu, Zhenya Huang, Chaokun Wang, Xunpeng Huang, Chengxiang Zhai, Enhong Chen
First submitted to arxiv on: 11 Dec 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Medium Difficulty summary: Large language models (LLMs) have excelled in various mathematical reasoning tasks due to their ability to learn contextually. However, few-shot demonstrations sometimes negatively impact LLMs’ performance, making their effectiveness unreliable. To address this limitation, we theoretically analyze the influence of in-context demonstrations on LLMs’ reasoning abilities and develop a demonstration selection method named LMS3. Our proposed approach leverages semantic similarity and inference stability to select relevant samples for different LLMs, incorporating a novel rejection mechanism to filter out unsuitable few-shot learning samples. Experimental results on three benchmarks, two LLM backbones, and multiple settings demonstrate the superiority of LMS3, achieving consistent improvements across all datasets, whereas existing methods have been unable to accomplish. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Low Difficulty summary: This paper looks at how well big language models can do math problems when shown a few examples. These models are good at learning from context, but sometimes showing them too many examples actually makes them worse. The researchers want to understand why this happens and find ways to make the models better. They came up with a new method called LMS3 that helps choose which examples to show the model, making sure they’re relevant and helpful. They tested it on different math problems and models and found that it works really well. |
Keywords
» Artificial intelligence » Few shot » Inference
