Summary of Math Neurosurgery: Isolating Language Models’ Math Reasoning Abilities Using Only Forward Passes, by Bryan R. Christ et al.
Math Neurosurgery: Isolating Language Models’ Math Reasoning Abilities Using Only Forward Passes
by Bryan R. Christ, Zack Gottesman, Jonathan Kropko, Thomas Hartvigsen
First submitted to arxiv on: 22 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper explores the encoding of math reasoning within Large Language Models (LLMs) and whether it can be isolated without affecting non-math behavior. The authors introduce Math Neurosurgery (MathNeuro), a method that efficiently isolates math-specific parameters in LLMs using forward passes. By pruning these identified parameters, they demonstrate a 4-17% improvement in math performance on GSM8K and 5-35% on MATH without affecting general language ability. The approach is also data-efficient, requiring only one sample to identify math-specific parameters. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Math reasoning is an important aspect of artificial intelligence that has implications for math education. This paper investigates how LLMs encode math reasoning and whether it can be isolated within models. The authors develop a new method called Math Neurosurgery (MathNeuro) that efficiently isolates math-specific parameters in LLMs. They show that by pruning these identified parameters, they can improve math performance without affecting general language ability. |
Keywords
» Artificial intelligence » Pruning
