Summary of Adversarial Math Word Problem Generation, by Roy Xie et al.
Adversarial Math Word Problem Generation
by Roy Xie, Chengxuan Huang, Junlin Wang, Bhuwan Dhingra
First submitted to arxiv on: 27 Feb 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a novel approach to ensuring fair evaluation in educational settings where large language models (LLMs) are prevalent. Current plagiarism detection tools struggle to keep pace with LLMs’ rapid advancements, making it challenging to assess students’ problem-solving abilities. The authors propose generating adversarial examples that preserve the structure and difficulty of original questions but are unsolvable by LLMs. Specifically, they focus on math word problems and use abstract syntax trees to structurally generate adversarial examples that cause LLMs to produce incorrect answers by editing numeric values. Experimental results on various open- and closed-source LLMs demonstrate that this method significantly degrades their math problem-solving ability. The authors also identify shared vulnerabilities among LLMs and propose a cost-effective approach to attack high-cost models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine trying to cheat on a math test with the help of a super smart computer program. That’s what large language models can do, but they’re making it hard for teachers to figure out if students are really understanding math or just using the computer to get the answers. To fix this problem, researchers came up with a new way to create fake math problems that these computers can’t solve. They used special tree-like structures to change the numbers in the problems, making them impossible for the computers to answer correctly. The results show that this method works well and can help teachers focus on what students really know. |
Keywords
» Artificial intelligence » Syntax
