MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation

Abstract

Emotion recognition in conversation (ERC) is a crucial component in affective dialogue systems, which helps the system understand users' emotions and generate empathetic responses. However, most works focus on modeling speaker and contextual information primarily on the textual modality or simply leveraging multimodal information through feature concatenation. In order to explore a more effective way of utilizing both multimodal and long-distance contextual information, we propose a new model based on multimodal fused graph convolutional network, MMGCN, in this work. MMGCN can not only make use of multimodal dependencies effectively, but also leverage speaker information to model inter-speaker and intra-speaker dependency. We evaluate our proposed model on two public benchmark datasets, IEMOCAP and MELD, and the results prove the effectiveness of MMGCN, which outperforms other SOTA methods by a significant margin under the multimodal conversation setting.

M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database

“…DialogueGCN (Ghosal et al, 2019) captures conversational dependencies between utterances with a graph-based structure. MMGCN (Hu et al, 2021) further proposes a GCNbased multimodal fusion method for multimodal ERC tasks to improve recognition performance. Di-alogXL (Shen et al, 2020) first introduces a strong pre-trained language model XLNet for text-based ERC.…”

“…MMGCN: A state-of-the-art GCN-based multimodal ERC framework proposed in (Hu et al, 2021). For the uni-modal experiments, we only model the fully connected graph.…”

“…(Poria et al, 2019b;Scherer, 2005;Koval et al, 2015). It has been proved in recent works Ghosal et al, 2019;Hu et al, 2021;Shen et al, 2020) that contextual information plays an important role in ERC tasks and brings significant improvements over baselines that only consider isolated utterances. DialogueRNN uses recurrent networks to model global and speaker-specific temporal-context information.…”

“…DialogueRNN uses recurrent networks to model global and speaker-specific temporal-context information. Di-alogueGCN (Ghosal et al, 2019) and MMGCN (Hu et al, 2021) use graph-based networks to capture conversational dependencies between utterances in dialogues. DialogXL (Shen et al, 2020) arXiv:2205.10237v1 [cs.CL] 9 May 2022 applies a strong pre-trained language model XLNet (Yang et al, 2019) to ERC and proposes a dialogaware self-attention method for modeling the context information.…”

M-MELD: A Multilingual Multi-Party Dataset for Emotion Recognition in Conversations

“…Moreover, DAG-ERC also makes meaningful and reasonable assumptions while constructing the graph by 1) Removing the link of an utterance in a dialogue to future utterances and 2) By imputing remote information for modeling conversational context by introducing another edge to the speakers previous utterance. Very recently MMGCN [15] proposed fusing information from multiple modalities by the use of spectral domain GCN to encode the multimodal contextual information. The work closest to our work is [11], where the authors use discourse relations between utterances to build a conversational graph and show that ER in both multi-party and two-party conversations benefit from conversational discourse structures.…”