Compact Graph Architecture for Speech Emotion Recognition

Shirian, Amir; Guha, Tanaya

doi:10.48550/arxiv.2008.02063

Cited by 2 publications

(1 citation statement)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Graph convolutional networks (GCNs) (Kipf and Welling 2016), which have been successfully used to address various problems in computer vision (Shen et al 2018) and natural language processing (Yao, Mao, and Luo 2019; Nan et al 2020), provide us with a new solution to address the emotion recognition problem. Several studies have applied the GCN to speech emotion recognition works (Shirian and Guha 2020;Ghosal et al 2019). However, they are only useful for unimodal tasks.…”

Section: Graph Neural Networkmentioning

confidence: 99%

Fusion with Hierarchical Graphs for Mulitmodal Emotion Recognition

Tang,

Luo,

Nan

et al. 2021

Preprint

View full text Add to dashboard Cite

Automatic emotion recognition (AER) based on enriched multimodal inputs, including text, speech, and visual clues, is crucial in the development of emotionally intelligent machines. Although complex modality relationships have been proven effective for AER, they are still largely underexplored because previous works predominantly relied on various fusion mechanisms with simply concatenated features to learn multimodal representations for emotion classification. This paper proposes a novel hierarchical fusion graph convolutional network (HFGCN) model that learns more informative multimodal representations by considering the modality dependencies during the feature fusion procedure. Specifically, the proposed model fuses multimodality inputs using a twostage graph construction approach and encodes the modality dependencies into the conversation representation. We verified the interpretable capabilities of the proposed method by projecting the emotional states to a 2D valence-arousal (VA) subspace. Extensive experiments showed the effectiveness of our proposed model for more accurate AER, which yielded state-of-the-art results on two public datasets, IEMOCAP and MELD.

show abstract