Deep Hierarchical Fusion with Application in Sentiment Analysis

Abstract

Recognizing the emotional tone in spoken language is a challenging research problem that requires modeling not only the acoustic and textual modalities separately but also their crossinteractions. In this work, we introduce a hierarchical fusion scheme for sentiment analysis of spoken sentences. Two bidirectional Long-Short-Term-Memory networks (BiLSTM), followed by multiple fully connected layers, are trained in order to extract feature representations for each of the textual and audio modalities. The representations of the unimodal encoders are both fused at each layer and propagated forward, thus achieving fusion at the word, sentence and high/sentiment levels. The proposed approach of deep hierarchical fusion achieves stateof-the-art results for sentiment analysis tasks. Through an ablation study, we show that the proposed fusion method achieves greater performance gains over the unimodal baseline compared to other fusion approaches in the literature.

Gated Mechanism for Attention Based Multi Modal Sentiment Analysis

“…[5] 76.5 73. 4 Zadeh et al [7] 76.9 77.0 Georgiou et al [9] 76.9 76.9 Poria et al [2] 77.64 -Ghosal et al [10] 82.31 80.69 Ghosal et al [10] 79.80 -Sun et al [4] 80 [7], ( ¦ ) results are obtained on CMU-MOSEI dataset after excluding the utterances with sentiment score of 0. We mention the results of proposed model with this setup in the parenthesis.…”

“…Methods that jointly learn the interactions between two or three modalities [3,4], and 3. Methods that explicitly learn contributions from these unimodal and cross modal cues, typically using attention based techniques [5,6,7,8,9,10].…”

“…Most of the existing approaches propose either fusion at different granularities [3,9] or use a cross interaction block that couple the features from different modalities [10,6].…”

A Two-Stage Attention Based Modality Fusion Framework for Multi-Modal Speech Emotion Recognition

“…In contrast, the early fusion could model inter-actions across modalities at raw features stage. Georgiou et al [4] concatenated features from different modalitiy at various levels and used multi-layer perceptron for emotion prediction. Generally speaking, concatenation based early fusion methods do not outperform the late fusion methods in SER [5].…”

What BERT Based Language Model Learns in Spoken Transcripts: An Empirical Study

“…The general modus-operandi of SLU systems is to convert voice into text using an ASR engine and use natural language understanding (NLU) on the transcribed text by modelling conversational and channel properties while being robust to ASR errors. Since spoken conversation boasts of amalgamation of spontaneous speaker interactions, it has become imperative for model architectures to capture multimodal features from text and speech modalities (Georgiou et al, 2019). The aim of these multimodal systems is to capture acoustic information such as pitch, intonation, rate of speech, etc.…”