Summary of Pandora: Towards General World Model with Natural Language Actions and Video States, by Jiannan Xiang et al.
Pandora: Towards General World Model with Natural Language Actions and Video States
by Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, Zhengzhong Liu, Eric P. Xing, Zhiting Hu
First submitted to arxiv on: 12 Jun 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces Pandora, a hybrid autoregressive-diffusion model that simulates future states of the world in response to different actions. This model enables interactive content creation and provides a foundation for grounded, long-horizon reasoning. The current foundation models, such as large language models (LLMs) and video models, are limited by their reliance on language modality or lack of control over simulated worlds. Pandora addresses these limitations by integrating a pretrained LLM and a pretrained video model, requiring only additional lightweight finetuning. This approach achieves domain generality, video consistency, and controllability through large-scale pretraining and instruction tuning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Pandora is a new way to create simulations of the world that can be controlled in real-time. Right now, there are two main types of models: language models that are great at understanding text but not so good at understanding images or actions, and video models that are great at generating videos but don’t understand text very well. Pandora combines these two types of models to create a new kind of model that can understand both text and images, and control the simulation in real-time. |
Keywords
» Artificial intelligence » Autoregressive » Diffusion model » Instruction tuning » Pretraining
