Getting the subtext without the text: Scalable multimodal sentiment classification from visual and acoustic modalities

Abstract

In the last decade, video blogs (vlogs) have become an extremely popular method through which people express sentiment. The ubiquitousness of these videos has increased the importance of multimodal fusion models, which incorporate video and audio features with traditional text features for automatic sentiment detection. Multimodal fusion offers a unique opportunity to build models that learn from the full depth of expression available to human viewers. In the detection of sentiment in these videos, acoustic and video features provide clarity to otherwise ambiguous transcripts. In this paper, we present a multimodal fusion model that exclusively uses high-level video and audio features to analyze spoken sentences for sentiment. We discard traditional transcription features in order to minimize human intervention and to maximize the deployability of our model on at-scale real-world data. We select high-level features for our model that have been successful in nonaffect domains in order to test their generalizability in the sentiment detection domain. We train and test our model on the newly released CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset, obtaining an F 1 score of 0.8049 on the validation set and an F 1 score of 0.6325 on the held-out challenge test set. Figure 1: A blindspot in multimodal sentiment analysis is the inclusion of human-transcriptions of spoken sentiment, which limits model applicability. We address this by using only prosodic and visual features for sentiment classification.

Leveraging Multimodal Behavioral Analytics for Automated Job Interview Performance Assessment and Feedback

“…The Maximum Entropy classifier was designed to integrate audio and text processing into a single system, and this model outperformed other conventional classifiers. Blanchard et al (2018) developed a fusion technique for audio and video modalities using audio and video features to analyze spoken sentences for the sentiment. They did not consider the traditional transcription features to minimize human intervention.…”

Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis

“…Authors evaluated their approach on the different datasets and reported improved accuracies in the range of 2-3% over the state-of-the-art models. Blanchard et al (2018) proposed a multi-modal fusion model that exclusively uses high-level visual and acoustic features for sentiment classification.…”

“…We compare our proposed approach against various existing systems (Nojavanasghari et al, 2016;Rajagopalan et al, 2016;Zadeh et al, , 2018aBlanchard et al, 2018) that made use of the same datasets. A comparative study is shown in Table 7.…”

Context-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis

“…For each dataset, we report three best systems for the comparisons 2 . In particular, we compare with the following systems: Bag of Feature -Multimodal Sentiment Analysis (BoF-MSA) (Blanchard et al, 2018), Memory Fusion Network (MFN) (Zadeh et al, 2018b), Deep Fusion -Deep Neural Network (DF-DNN) (Nojavanasghari et al, 2016), Multi View -LSTM (MV-LSTM) (Rajagopalan et al, 2016), Early Fusion -LSTM (EF-LSTM) (Zadeh et al, 2018c), Tensor Fusion Network (TFN) , Random Forest (RF) (Breiman, 2001), Support Vector Machine (Zadeh et al, 2016), Multi-Attention Recurrent Network (MARN) (Zadeh et al, 2018a), Dynamic Fusion Graph (DFG) (Zadeh et al, 2018c), Multi Modal Multi Utterance-Bimodal Attention (MMMU-BA) (Ghosal et al, 2018), Bi-directional Contextual LSTM (BC-LSTM) (Poria et al, 2017b) and Multimodal Factorization Model (MFM) (Tsai et al, 2018).…”

“…Poria et al (2017a) presented a literature survey on various affect dimensions e.g., sentiment analysis, emotion analysis, etc., for the multi-modal analysis. A multi-modal fusion-based approach is proposed in (Blanchard et al, 2018) for sentiment classification. The author used exclusively high-level fusion of visual and acoustic features to classify the sentiment.…”