Summary of Dynamic Modality and View Selection For Multimodal Emotion Recognition with Missing Modalities, by Luciana Trinkaus Menon et al.
Dynamic Modality and View Selection for Multimodal Emotion Recognition with Missing Modalities
by Luciana Trinkaus Menon, Luiz Carlos Ribeiro Neduziak, Jean Paul Barddal, Alessandro Lameiras Koerich, Alceu de Souza Britto Jr
First submitted to arxiv on: 18 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD); Audio and Speech Processing (eess.AS)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents a significant advancement in multimodal emotion recognition (MER) using artificial intelligence (AI). It addresses the challenge of AI models dealing with the absence of a particular modality, which is common in real-world situations. The study focuses on assessing the performance and resilience of two strategies: a novel multimodal dynamic modality and view selection, and a cross-attention mechanism. Experimental results on the RECOLA dataset demonstrate that dynamic selection-based methods are promising for MER, outperforming the baseline even when one modality is missing. This highlights the importance of understanding the interplay between audio and video modalities in emotion prediction. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Artificial intelligence (AI) is getting better at recognizing human emotions. Traditionally, this was a job for psychologists and neuroscientists, but AI has changed that. There are many ways to understand emotions, like listening to someone’s voice or reading their facial expressions. However, AI still faces some big challenges when it comes to understanding emotions from multiple sources. One of the main problems is what happens when one of these sources is missing – like when you can’t hear someone’s voice because they’re wearing a mask. This study looks at how two different approaches work in this situation: using dynamic modality and view selection, or cross-attention mechanisms. The results show that both methods do better than usual when one source is missing. |
Keywords
» Artificial intelligence » Cross attention » Mask
