Summary of Can-do! a Dataset and Neuro-symbolic Grounded Framework For Embodied Planning with Large Multimodal Models, by Yew Ken Chia et al.
Can-Do! A Dataset and Neuro-Symbolic Grounded Framework for Embodied Planning with Large Multimodal Models
by Yew Ken Chia, Qi Sun, Lidong Bing, Soujanya Poria
First submitted to arxiv on: 22 Sep 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The abstract introduces Can-Do, a benchmark dataset designed to evaluate embodied planning abilities in realistic environments. The dataset consists of 400 multimodal samples, including natural language user instructions, visual images, state changes, and corresponding action plans. State-of-the-art models face bottlenecks in visual perception, comprehension, and reasoning abilities. To address these challenges, the authors propose NeuroGround, a neurosymbolic framework that grounds plan generation in perceived environment states and leverages symbolic planning engines to augment model-generated plans. Experimental results demonstrate the effectiveness of this framework compared to strong baselines. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Can-Do is a new way to test how well computers can make plans and take actions in different situations. This dataset has lots of examples, each with instructions from a person, pictures of what’s happening, changes that happen, and what steps to take next. Computers have trouble seeing things, understanding, and making good choices. To help them, the researchers created NeuroGround, which helps computers make better plans by using what they’ve seen and some special planning tools. The results show that this new approach works better than other ways of doing it. |
