Summary of Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations, by Sangmin Lee et al.
Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations
by Sangmin Lee, Bolin Lai, Fiona Ryan, Bikram Boote, James M. Rehg
First submitted to arxiv on: 4 Mar 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces three new tasks to model the dynamics of multi-party social interactions, including speaking target identification, pronoun coreference resolution, and mentioned player prediction. The authors contribute extensive data annotations in social deduction game settings and propose a novel multimodal baseline that leverages densely aligned language-visual representations. This approach facilitates capturing verbal and non-verbal cues pertinent to social reasoning, demonstrating its effectiveness in modeling fine-grained social interactions. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about understanding how people interact with each other through words and body language. Most research on this topic only looks at one person or a group as a whole, but this paper tries to model the complex dynamics of multiple people talking together. The authors create new challenges for machines to learn from, including identifying who’s speaking, figuring out what pronouns refer to, and predicting what someone is trying to say. They also propose a way for machines to process language and images together more effectively, which helps them better understand social interactions. |
Keywords
* Artificial intelligence * Coreference
