An Ensemble Approach to Utterance Level Multimodal Sentiment Analysis

Huddar, Mahesh G.; Sannakki, Sanjeev S.; Rajpurohit, Vijay S.

doi:10.1109/ctems.2018.8769162

Cited by 13 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Contributions. While most of the current works [9,10,11,12] neglect the indispensable role of the cognitive cues in multimodal sentiment analysis, our proposed framework in this paper enhances analytical results by leveraging userspecific latent characteristics. To the best of our knowledge, we reveal the utopian spirit of the hidden parameters and devise the first approach based on an adaptive tree that utilizes an attention-based fusion to facilitate cognitiveoriented knowledge transfer within the tree, forming an ensemble of submodels.…”

Section: Fig 1: Base Models Accuracy Comparisonmentioning

confidence: 99%

Transfer-based adaptive tree for multimodal sentiment analysis based on user latent aspects

Rahmani¹,

Hosseini²,

Zall³

et al. 2021

Preprint

View full text Add to dashboard Cite

Multimodal sentiment analysis benefits various applications such as human-computer interaction and recommendation systems. It aims to infer the users' bipolar ideas using visual, textual, and acoustic signals. Although researchers affirm the association between cognitive cues and emotional manifestations, most of the current multimodal approaches in sentiment analysis disregard user-specific aspects. To tackle this issue, we devise a novel method to perform multimodal sentiment prediction using cognitive cues, such as personality. Our framework constructs an adaptive tree by hierarchically dividing users and trains the LSTM-based submodels, utilizing an attention-based fusion to transfer cognitive-oriented knowledge within the tree. Subsequently, the framework consumes the conclusive agglomerative knowledge from the adaptive tree to predict final sentiments. We also devise a dynamic dropout method to facilitate data sharing between neighboring nodes, reducing data sparsity. The empirical results on real-world datasets determine that our proposed model for sentiment prediction can surpass trending rivals. Moreover, compared to other ensemble approaches, the proposed transfer-based algorithm can better utilize the latent cognitive cues and foster the prediction outcomes. Based on the given extrinsic and intrinsic analysis results, we note that compared to other theoretical-based techniques, the proposed hierarchical clustering approach can better group the users within the adaptive tree.

show abstract

Section: Fig 1: Base Models Accuracy Comparisonmentioning

confidence: 99%

Transfer-based adaptive tree for multimodal sentiment analysis based on user latent aspects

Rahmani¹,

Hosseini²,

Zall³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The sentiment analysis and affective computing have a lot of challenges such as subjectivity analysis, aspect detection and consideration, topic detection and tracking, document summarization . The earlier research concentrates on textual data for sentiment analysis but more recently multimodal data such as textual‐acoustic‐visual modality were considered for sentiment classification and affective computing . Usually, handcrafted features or lexicons or networks or ontologies are used in sentiment classification.…”

Section: Literature Reviewmentioning

confidence: 99%

Multi‐level feature optimization and multimodal contextual fusion for sentiment analysis and emotion classification

Huddar¹,

Sannakki²,

Rajpurohit³

2020

Computational Intelligence

Self Cite

View full text Add to dashboard Cite

The availability of the humongous amount of multimodal content on the internet, the multimodal sentiment classification, and emotion detection has become the most researched topic. The feature selection, context extraction, and multi‐modal fusion are the most important challenges in multimodal sentiment classification and affective computing. To address these challenges this paper presents multilevel feature optimization and multimodal contextual fusion technique. The evolutionary computing based feature selection models extract a subset of features from multiple modalities. The contextual information between the neighboring utterances is extracted using bidirectional long‐short‐term‐memory at multiple levels. Initially, bimodal fusion is performed by fusing a combination of two unimodal modalities at a time and finally, trimodal fusion is performed by fusing all three modalities. The result of the proposed method is demonstrated using two publically available datasets such as CMU‐MOSI for sentiment classification and IEMOCAP for affective computing. Incorporating a subset of features and contextual information, the proposed model obtains better classification accuracy than the two standard baselines by over 3% and 6% in sentiment and emotion classification, respectively.

show abstract

“…In the early literature low-level handcrafted (manually extracted) features such as lexicon representation for textual data, low-level descriptors for speech and facial expressions for video modalities (Rosas et al, 2013) or ontologies (Mahmoud et al, 2018) or sentiment lexicons (Mohammad et al, 2013) were considered. Instead of simplistic fusion techniques such as feature level fusion (Pérez- Rosas et al, 2013) or model-based fusion (Du et al, 2018) or decision level fusion (Huddar et al, 2018), shallow fusion techniques were proposed to understand the complex correlation between modalities.…”

Section: Related Workmentioning

confidence: 99%

“…Recent literature in multimodal affective computing uses either early fusion (also known as feature concatenation) (Pérez- Rosas et al, 2013) or model-based fusion (Du et al, 2018) or decision fusion (also known as late fusion) (Huddar et al, 2018). In early or feature level fusion, features from different modalities are concatenated.…”

Section: Problem With Early Model-based and Late Fusionmentioning

confidence: 99%

“…The combination of different audio-visual-textual modalities helps us to build more accurate and robust multimodal affective computing models (Huddar et al, 2019a). Basically, sentiment analysis and emotion classification models extract features from the audio-visual modalities, use either feature (early) level fusion (Pérez- Rosas et al, 2013) or model-based fusion (Du et al, 2018) or decision (late) level fusion (Huddar et al, 2018) to merge the information extracted from audiovisual modalities.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Attention-based word-level contextual feature extraction and cross-modality fusion for sentiment analysis and emotion classification

Huddar¹,

Sannakki²,

Rajpurohit³

2020

IJIEI

Self Cite

View full text Add to dashboard Cite

Multimodal affective computing has become a popular research area, due to the availability of a large amount of multimodal content. Feature alignment between the modalities and multimodal fusion are the most important issues in multimodal affective computing. To address these issues, the proposed model extracts the features at word-level and forced alignment is used to understand the time-dependent interaction among the modalities. The contextual information among the words of an utterance and between the nearby utterances is extracted using bidirectional long short term memory (LSTM). Weighted pooling based attention model is used to select the important features within the modalities and importance of each modality. Information from multiple modalities is fused using a cross-modality fusion technique. The performance of the proposed model was tested on two standard datasets such as IEMOCAP and CMU-MOSI. By incorporating the word-level features, feature alignment, and cross-modality fusion, the proposed architecture outperforms the baselines in terms of classification accuracy.

show abstract

An Ensemble Approach to Utterance Level Multimodal Sentiment Analysis

Cited by 13 publications

References 22 publications

Transfer-based adaptive tree for multimodal sentiment analysis based on user latent aspects

Transfer-based adaptive tree for multimodal sentiment analysis based on user latent aspects

Multi‐level feature optimization and multimodal contextual fusion for sentiment analysis and emotion classification

Attention-based word-level contextual feature extraction and cross-modality fusion for sentiment analysis and emotion classification

Contact Info

Product

Resources

About