Summary of Language Models Are Symbolic Learners in Arithmetic, by Chunyuan Deng et al.
Language Models are Symbolic Learners in Arithmetic
by Chunyuan Deng, Zhiqi Li, Roy Xie, Ruidi Chang, Hanjie Chen
First submitted to arxiv on: 21 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Large Language Models (LLMs) are thought to struggle with arithmetic learning due to differences between language modeling and numerical computation, but concrete evidence has been lacking. This work investigates whether LLMs leverage partial products during arithmetic learning and find that they can identify some partial products after learning, but fail to use them for arithmetic tasks. The paper also explores how LLMs approach arithmetic symbolically by breaking tasks into subgroups, hypothesizing that difficulties arise from subgroup complexity and selection. Results show that when subgroup complexity is fixed, LLMs treat a collection of different arithmetic operations similarly. Analysis reveals a U-shaped pattern in position-level accuracy across different training sizes, suggesting that LLMs select subgroups following an easy-to-hard paradigm during learning. This study confirms that LLMs are pure symbolic learners in arithmetic tasks and highlights the importance of understanding them deeply through subgroup-level quantification. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large Language Models struggle with math, but why? Researchers investigated whether these models use partial products when doing math problems. They found that although the models can identify some partial products, they don’t actually use them to solve math problems. The team also looked at how the models do math symbolically by breaking down tasks into smaller groups. Surprisingly, the models treat all the different math operations in each group similarly. As the models learn more, they start doing better on harder math problems in the middle of each group. This study shows that Large Language Models are really good at doing math, but only because they’re using a special trick to make it easier. |
