Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis

Abstract

Abstract-Technology has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is multimodal. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of video on the Internet will only continue to increase. It has become increasingly difficult for researchers to keep up with this deluge of multimodal content, let alone organize or make sense of it. Mining useful knowledge from video is a critical need that will grow exponentially, in pace with the global growth of content. This is particularly important in sentiment analysis, as both service and product reviews are gradually shifting from unimodal to multimodal. We present a novel method to extract features from visual and textual modalities using deep convolutional neural networks. By feeding such features to a multiple kernel learning classifier, we significantly outperform the state of the art of multimodal emotion recognition and sentiment analysis on different datasets.

Construction of Spontaneous Emotion Corpus from Indonesian TV Talk Shows and Its Application on Multimodal Emotion Recognition

“…In a study using an acted emotion corpus, Busso et al reported an accuracy of 89% in recognizing four classes of emotion, a significant improvement compared to the accuracy of 70.9% with speech-based system and 85% with facial expression based system [37]. This finding is reaffirmed by Poria et al who reported improvement when fusing three modalities into a single emotion recognizer [38]. Another study by Nojavanasghari et al focusing on spontaneous emotions in children also reported the same trend, with best overall performance at 69% [39].…”

Context-Dependent Sentiment Analysis in User-Generated Videos

“…Poria et al (Poria et al, 2015(Poria et al, , 2016d) extracted audio, visual and textual features using convolutional neural network (CNN); concatenated those features and employed multiple kernel learning (MKL) for final sentiment classification. (Metallinou et al, 2008) and (Eyben et al, 2010a) fused audio and textual modalities for emotion recognition.…”

Sentiment Analysis, Basic Tasks of

“…This allows analysts in government, commercial and political domains who need to determine the response of people to different crisis events [5,40,59]. Similarly, online reviews need to be summarized in a manner that allows comparison of opinions, so that a user can clearly see the advantages and weaknesses of each product merely with a single glance, both in unimodal [60] and multimodal [50,9] contexts. Further, we can do in-depth opinion assessment, such as finding reasons or aspects [46] in opinion-bearing texts.…”