Multi-Modal Audio, Video and Physiological Sensor Learning for Continuous Emotion Prediction

Brady, Kevin; Gwon, Youngjune; Khorrami, Pooya; Godoy, Elizabeth; Campbell, William M.; Dagli, Charlie K.; Huang, Thomas S.

doi:10.1145/2988257.2988264

Cited by 96 publications

(70 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These trends are exemplified in the annual competitions Emotion Recognition in the Wild (EmotiW) [12] and Audio Video Emotion Challenge (AVEC) [13]. Since 2010, deep learning methods have been applied to affect recognition problems across multiple modalities and led to improvements in accuracy, including winning performances at EmotiW [14], [15], [16] and AVEC [17], [18], [19].…”

Section: Introductionmentioning

confidence: 99%

Deep Learning for Human Affect Recognition: Insights and New Developments

Rouast

Adam

Chiong

2021

IEEE Trans. Affective Comput.

147

View full text Add to dashboard Cite

Automatic human affect recognition is a key step towards more natural human-computer interaction. Recent trends include recognition in the wild using a fusion of audiovisual and physiological sensors, a challenging setting for conventional machine learning algorithms. Since 2010, novel deep learning algorithms have been applied increasingly in this field. In this paper, we review the literature on human affect recognition between 2010 and 2017, with a special focus on approaches using deep neural networks. By classifying a total of 950 studies according to their usage of shallow or deep architectures, we are able to show a trend towards deep learning. Reviewing a subset of 233 studies that employ deep neural networks, we comprehensively quantify their applications in this field. We find that deep learning is used for learning of (i) spatial feature representations, (ii) temporal feature representations, and (iii) joint feature representations for multimodal sensor data. Exemplary state-of-the-art architectures illustrate the progress. Our findings show the role deep architectures will play in human affect recognition, and can serve as a reference point for researchers working on related applications.

show abstract

Section: Introductionmentioning

confidence: 99%

Deep Learning for Human Affect Recognition: Insights and New Developments

Rouast

Adam

Chiong

2021

IEEE Trans. Affective Comput.

147

View full text Add to dashboard Cite

show abstract

“…One limitation of our current LSTM models is that we do not leverage the ability of neural networks to extract features directly from the raw data. For example, many previous models use a CNN on the raw images to extract visual features (e.g., [36], [37]), rather than calculating visual features separately as we did here. The weights of such a CNN will be modified during training, which "optimizes" the feature extraction process for this particular task.…”

Section: Lstm Resultsmentioning

confidence: 99%

“…Many researchers have since used RNNs and their LSTM variants to recognize emotion from speech and from video. [36], [37], [38] and [25] all used a Convolutional Neural Network to learn hidden layer features from individual video frames, along with a recurrency between hidden layers at consecutive times-thus, combining the time-independent CNN with a RNN. Many others have used LSTMs to recognize emotions from video data.…”

Section: Discriminative Modelsmentioning

confidence: 99%

“…One popular method (and the one that we use in this paper) is feature fusion, also called early fusion, where the input modalities are oversampled, undersampled, or otherwise averaged, to a common sampling rate. This allows the multimodal features to be concatenated into a single feature vector within a given time window, to be fed into a model [37], [42], [43]. A second way to achieve such "synchronization" is decision fusion (or late fusion): This involves fitting a separate time-series model to each modality, operating at their own sampling frequencies.…”

Section: Discriminative Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset

Ong

Zhi-Xuan

et al. 2021

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

Human emotions unfold over time, and more affective computing research has to prioritize capturing this crucial component of real-world affect. Modeling dynamic emotional stimuli requires solving the twin challenges of time-series modeling and of collecting high-quality time-series datasets. We begin by assessing the state-of-the-art in time-series emotion recognition, and we review contemporary time-series approaches in affective computing, including discriminative and generative models. We then introduce the first version of the Stanford Emotional Narratives Dataset (SENDv1): a set of rich, multimodal videos of self-paced, unscripted emotional narratives, annotated for emotional valence over time. The complex narratives and naturalistic expressions in this dataset provide a challenging test for contemporary time-series emotion recognition models. We demonstrate several baseline and state-of-the-art modeling approaches on the SEND, including a Long Short-Term Memory model and a multimodal Variational Recurrent Neural Network, which perform comparably to the human-benchmark. We end by discussing the implications for future research in time-series affective computing.recognition. Specifically, we define time-series modeling as taking in temporally continuous input data and producing temporally continuous output, with an explicit consideration of how information is propagated over time. For instance, in order to engage in such inference, a social robot in conversation with its user would have to take in a continuous stream of sensor data, process them, and reason about their user's emotions over time, perhaps after every second or after every sentence, as well as across many sentences in the conversation and across multiple conversations [11].Despite the progress that has been made in time-series emotion recognition in the past decade, the field is still far from affective robots that can understand human emotions in daily life. What is needed to achieve this ambitious goal? We suggest that the biggest barriers to overcome are due to (1) the inherent difficulty of building computational time-series models, and (2) the difficulty of collecting highquality datasets. To address this first gap, we conduct a review covering different machine-learning-based approaches to time-series modeling (Section 2). We begin by discussing the most common time-series techniques in affective computing: deep neural network models, part of a broader class of discriminative models. We also cover generative time-series approaches, which are comparatively less popular within affective computing, but offer interesting modeling capabilities and hold exciting potential for emotion understanding.We turn next to discuss the second gap: Researchers need high-quality time-series datasets on which to train models. These are expensive to construct, in terms of both the production of stimuli and the collection of timeseries annotations of emotion and affective labeling [12]. There are several existing time-series datasets that have been used by the ...

show abstract

“…In addition, the LSTM model has low performance for predicting movie induced emotions [31], 2 yet it has achieved leading performance in various emotion recognition tasks due to its ability to model temporal context (e.g. [32]). Ma et al [31] predict movie induced emotions at an interval of 10 seconds, which already contains temporal context.…”

Section: Previous Work On Liris-accede Databasementioning

confidence: 99%

Recognizing induced emotions of movie audiences: Are induced and perceived emotions the same?

Tian

Muszyński

Lai

et al. 2017

2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)

View full text Add to dashboard Cite

Abstract-Predicting the emotional response of movie audiences to affective movie content is a challenging task in affective computing. Previous work has focused on using audiovisual movie content to predict movie induced emotions. However, the relationship between the audience's perceptions of the affective movie content (perceived emotions) and the emotions evoked in the audience (induced emotions) remains unexplored. In this work, we address the relationship between perceived and induced emotions in movies, and identify features and modelling approaches effective for predicting movie induced emotions. First, we extend the LIRIS-ACCEDE database by annotating perceived emotions in a crowd-sourced manner, and find that perceived and induced emotions are not always consistent. Second, we show that dialogue events and aesthetic highlights are effective predictors of movie induced emotions. In addition to movie based features, we also study physiological and behavioural measurements of audiences. Our experiments show that induced emotion recognition can benefit from including temporal context and from including multimodal information. Our study bridges the gap between affective content analysis and induced emotion prediction.

show abstract

Multi-Modal Audio, Video and Physiological Sensor Learning for Continuous Emotion Prediction

Cited by 96 publications

References 23 publications

Deep Learning for Human Affect Recognition: Insights and New Developments

Deep Learning for Human Affect Recognition: Insights and New Developments

Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset

Recognizing induced emotions of movie audiences: Are induced and perceived emotions the same?

Contact Info

Product

Resources

About