Summary of Omnijarvis: Unified Vision-language-action Tokenization Enables Open-world Instruction Following Agents, by Zihao Wang et al.
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents
by Zihao Wang, Shaofei Cai, Zhancun Mu, Haowei Lin, Ceyao Zhang, Xuejie Liu, Qing Li, Anji Liu, Xiaojian Ma, Yitao Liang
First submitted to arxiv on: 27 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel Vision-Language-Action (VLA) model, OmniJARVIS, is proposed for open-world instruction-following agents in Minecraft. Unlike prior works that emit textual goals or produce control commands directly, OmniJARVIS uses unified tokenization of multimodal interaction data to ensure strong reasoning and efficient decision-making capabilities. The model consists of a self-supervised behavior encoder and an imitation learning policy decoder conditioned on the encoded tokens. These tokens are then used to pack long-term interactions into unified sequences, which are modeled with autoregressive transformers. OmniJARVIS demonstrates excellent performances on various tasks in open-world Minecraft, including atomic, programmatic, and open-ended tasks. The model’s performance is attributed to its ability to reason, plan, answer questions, and act through the use of semantically meaningful behavior tokens. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A new AI model called OmniJARVIS can understand and follow instructions in a game called Minecraft. This model is special because it uses information from different senses (like vision and language) to make decisions. The model works by first understanding what someone wants, then planning how to do it, and finally doing the action. This process happens quickly and efficiently thanks to the way the model organizes information into “tokens.” OmniJARVIS can perform many tasks in Minecraft, from simple ones to more complex ones. |
Keywords
» Artificial intelligence » Autoregressive » Decoder » Encoder » Self supervised » Tokenization
