Summary of Language Models Encode Numbers Using Digit Representations in Base 10, by Amit Arnold Levy et al.
Language Models Encode Numbers Using Digit Representations in Base 10
by Amit Arnold Levy, Mor Geva
First submitted to arxiv on: 15 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Large language models (LLMs) often struggle with simple numerical problems, such as comparing small numbers. This study investigates whether these errors arise from how LLMs represent numbers. Our findings suggest that LLMs internally represent numbers using circular representations for each digit in base 10, rather than a value representation. This unique representation sheds light on the error patterns of models when performing tasks involving numerical reasoning. We use probing experiments and causal interventions to demonstrate this phenomenon. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models (LLMs) sometimes make mistakes with simple math problems, like comparing small numbers. Researchers wanted to figure out why this happens. They discovered that LLMs represent numbers in a special way – each digit has its own circular representation, rather than representing the number’s value as we do. This helps explain why LLMs get some math problems wrong and could help us understand how they work better. |
