Summary of S3: a Simple Strong Sample-effective Multimodal Dialog System, by Elisei Rykov et al.
S3: A Simple Strong Sample-effective Multimodal Dialog System
by Elisei Rykov, Egor Malkershin, Alexander Panchenko
First submitted to arxiv on: 26 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents a simple yet powerful baseline for the multimodal dialog task, achieving near state-of-the-art results on two leaderboards: MMMU and AI Journey Contest 2023. The system is built around a pre-trained large language model, pre-trained modality encoders for image and audio, and a trainable modality projector. The proposed data mixture demonstrates that a multimodal model based on a strong language model and trained on a small amount of multimodal data can perform efficiently in the task. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper introduces a new approach to multimodal dialog that combines the strengths of language models and modality-specific encoders. By pre-training these components separately, the authors show that their system can achieve state-of-the-art results with minimal additional training data. This makes it an attractive solution for applications where large amounts of labeled data are not available. |
Keywords
» Artificial intelligence » Language model » Large language model
