Summary of Vc Search: Bridging the Gap Between Well-defined and Ill-defined Problems in Mathematical Reasoning, by Shi-yu Tian et al.
VC Search: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning
by Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang, Lin-Han Jia, Lan-Zhe Guo, Yu-Feng Li
First submitted to arxiv on: 7 Jun 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The abstract discusses the limitations of current methods for evaluating large language models (LLMs) on mathematical reasoning tasks. Existing benchmarks focus on carefully constructed problems and neglect real-world scenarios where conditions may be missing or contradictory. To address this issue, a new benchmark called Problems with Missing and Contradictory conditions (PMC) is developed, containing over 5,000 validated ill-defined mathematical problems. Preliminary experiments show that existing methods struggle to balance accuracy and rejection capabilities, while formal methods are challenged by modeling complex problems. A training-free framework called Variable-Constraint Search (VCSEARCH) is proposed to detect ill-defined problems using a variable-constraint pair search strategy. Experiments demonstrate that VCSEARCH improves the accuracy of identifying unsolvable problems by at least 12% across different LLMs, enhancing their robust mathematical reasoning ability. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper studies how well large language models (LLMs) can solve math problems when some information is missing or doesn’t make sense. Right now, we test these models on carefully made problems that don’t have any missing or contradictory conditions. But in real life, math problems often come with missing or conflicting information. To help LLMs get better at solving these kinds of problems, a new benchmark called PMC was created. It has over 5,000 validated math problems where some information is missing or doesn’t make sense. The results show that current methods are not very good at balancing how well they solve the problem and how often they say it’s too hard to solve. Formal methods also struggle with making models for these complex problems. To address this challenge, a new framework called VCSEARCH was developed. It uses formal language to detect when a problem is too hard or doesn’t make sense. The results show that VCSEARCH helps LLMs get better at solving math problems by at least 12% across different models. |
