Summary of Natural Plan: Benchmarking Llms on Natural Language Planning, by Huaixiu Steven Zheng et al.
NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
by Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V. Le, Ed H. Chi, Denny Zhou
First submitted to arxiv on: 6 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary NATURAL PLAN is a realistic planning benchmark in natural language that assesses large language models’ (LLMs) capabilities. It comprises three key tasks: Trip Planning, Meeting Planning, and Calendar Scheduling. The evaluation focuses on providing full information about the task using tools like Google Flights, Maps, and Calendar as contexts for LLMs. This eliminates the need for a tool-use environment. We observe that NATURAL PLAN is a challenging benchmark for state-of-the-art (SoTA) models. For example, GPT-4 and Gemini 1.5 Pro achieved only 31.1% and 34.8% solve rates in Trip Planning, respectively. Model performance drops significantly as problem complexity increases: all SoTA LLMs perform below 5% when there are 10 cities, highlighting a significant gap in planning in natural language for SoTA LLMs. The paper also conducts ablation studies on NATURAL PLAN to investigate the effectiveness of approaches like self-correction, few-shot generalization, and in-context planning with long-contexts on improving LLM planning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary NATURAL PLAN is a new way to test how well artificial intelligence models can plan things. It’s like asking Siri or Google Assistant to book a trip or schedule a meeting. We want to see if these AI models are good at planning and making decisions. To do this, we give them information about the task they need to complete, like flight schedules or calendar events. This helps us figure out how well they can plan without needing special tools. The results show that even the best AI models struggle with planning complex tasks. We also tried different ways to help these AI models improve their planning skills and found some things that worked better than others. |
Keywords
» Artificial intelligence » Few shot » Gemini » Generalization » Gpt
