Summary of Mllm As Retriever: Interactively Learning Multimodal Retrieval For Embodied Agents, by Junpeng Yue et al.
MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents
by Junpeng Yue, Xinru Xu, Börje F. Karlsson, Zongqing Lu
First submitted to arxiv on: 4 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed MART method utilizes interaction data to fine-tune an MLLM retriever based on preference learning, considering the effectiveness of trajectories for specific tasks. This enhances the performance of embodied agents in complex tasks by prioritizing task-relevant trajectory data. The novel Trajectory Abstraction mechanism represents trajectories with fewer tokens while preserving key information, enabling better comprehension of milestones. Experimental results demonstrate significant improvements in task success rates compared to baseline methods across various environments. This work presents a new paradigm for multimodal retrieval in embodied agents, leveraging MLLM’s summarization capabilities and preference learning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary MLLM agents can do complex tasks by getting the right information from trajectories. Right now, most methods focus on what surfaces look like or sound like, without thinking about how effective they are for the task. To fix this, we created a new method called MART that uses interaction data to make an MLLM retriever better at finding relevant information. We also made a Trajectory Abstraction mechanism that helps agents understand milestones in trajectories better. Our experiments showed that our method is much better than others at doing tasks in new situations. |
Keywords
» Artificial intelligence » Summarization
