Summary of From Specific-mllms to Omni-mllms: a Survey on Mllms Aligned with Multi-modalities, by Shixin Jiang et al.
From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities
by Shixin Jiang, Jiafeng Liang, Jiyuan Wang, Xuan Dong, Heng Chang, Weijiang Yu, Jinhua Du, Ming Liu, Bing Qin
First submitted to arxiv on: 16 Dec 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper surveys Omni-MLLMs, which aim to achieve omni-modal understanding and generation by mapping various non-linguistic modalities into the embedding space of large language models (LLMs). The authors provide a comprehensive overview of these models, discussing their four core components for unified multi-modal modeling. They also introduce effective integration techniques through two-stage training and evaluate them using relevant datasets. Challenges and future directions are outlined, making this paper a useful introduction to beginners in the field. The authors hope to promote advancements in related research. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Omni-MLLMs aim to understand and generate combinations of different types of data, like images, audio, or text. This paper explores these models, explaining how they work and what makes them special. It also looks at some examples of how they’ve been used and what challenges remain to be solved. |
Keywords
» Artificial intelligence » Embedding space » Multi modal
