Summary of Fcorebench: Can Large Language Models Solve Challenging First-order Combinatorial Reasoning Problems?, by Chinmay Mittal et al.
FCoReBench: Can Large Language Models Solve Challenging First-Order Combinatorial Reasoning Problems?
by Chinmay Mittal, Krishna Kartik, Mausam, Parag Singla
First submitted to arxiv on: 4 Feb 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, researchers investigate whether large language models (LLMs) can solve complex first-order combinatorial reasoning problems like graph coloring and cryptarithmetic. They present a new dataset called FCoReBench, which contains 40 challenging problems with varying sizes, along with scripts to generate and verify solutions. The study finds that LLMs perform poorly on this dataset, especially as problem size increases. To improve performance, the authors propose SymPro-LM, an approach that combines LLMs with symbolic solvers and program interpreters, providing feedback from a few solved examples. This method achieves significant gains in performance and is robust to changes in problem size. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Can computers solve tricky problems like coloring graphs or doing arithmetic? Researchers created a special dataset called FCoReBench to test large language models (LLMs) on these types of challenges. They found that LLMs struggled, especially with bigger problems. To help them do better, the team developed SymPro-LM, which combines LLMs with other problem-solving tools and uses previous solutions as examples. This new approach worked much better and could handle big problems. |
