Incorporating Interpersonal Synchronization Features for Automatic Emotion Recognition from Visual and Audio Data during Communication

Abstract

During social interaction, humans recognize others’ emotions via individual features and interpersonal features. However, most previous automatic emotion recognition techniques only used individual features—they have not tested the importance of interpersonal features. In the present study, we asked whether interpersonal features, especially time-lagged synchronization features, are beneficial to the performance of automatic emotion recognition techniques. We explored this question in the main experiment (speaker-dependent emotion recognition) and supplementary experiment (speaker-independent emotion recognition) by building an individual framework and interpersonal framework in visual, audio, and cross-modality, respectively. Our main experiment results showed that the interpersonal framework outperformed the individual framework in every modality. Our supplementary experiment showed—even for unknown communication pairs—that the interpersonal framework led to a better performance. Therefore, we concluded that interpersonal features are useful to boost the performance of automatic emotion recognition tasks. We hope to raise attention to interpersonal features in this study.

Novel Speech-Based Emotion Climate Recognition in Peers’ Conversations Incorporating Affect Dynamics and Temporal Convolutional Neural Networks

“…For the case of K-EmoCon comparative analysis (Table 2), the AffECt model outperformed the state-of-the-art ones [50], [53] by a considerable margin, demonstrating its effectiveness in EC arousal and valence classification. The highest accuracy achieved by the AffECt model on EC arousal [50], [53] (max of 75.1% (arousal) and 68.3% (valence) using CNN-Bi-LSTM [50]).…”

“…For the case of K-EmoCon comparative analysis (Table 2), the AffECt model outperformed the state-of-the-art ones [50], [53] by a considerable margin, demonstrating its effectiveness in EC arousal and valence classification. The highest accuracy achieved by the AffECt model on EC arousal [50], [53] (max of 75.1% (arousal) and 68.3% (valence) using CNN-Bi-LSTM [50]). Additionally, the highest sensitivity achieved by the AffECt model on EC arousal and valence classification using KNN is 91.9% and 92.5%, respectively, surpassing the highest sensitivity achieved by the other models [50], [53] (max of 79.2% (arousal) and 72.3% (valence) using CNN-BiLSTM [50]).…”

Mobile Emotion Recognition via Multiple Physiological Signals using Convolution-augmented Transformer

“…To avoid data leakage, we did not use data re-sampling approaches [24] to solve the imbalanced data problem as they will change the original dataset itself. Instead, we chose focal loss [16] as the loss function of our model, which can automatically downweight the contribution of easily classified samples and focus on hard misclassified samples by applying a modulating term to the cross-entropy loss.…”

“…Considering that the classes of arousal and valence are heavily imbalanced, we did not use recognition accuracy as the evaluation metric like most studies. Instead, following [17,24], we chose the average F1 score (Macro-F1) and unweighted average recall (UAR) as our validation metrics. These metrics give the same importance to each class, and are defined as the mean of class-wise F1 scores and recall scores respectively.…”

Analytics and Applications of Audio and Image Sensing Techniques

“…The study presented in [ 16 ] by J. Quan, Y. Miyake, and T. Nozawa, investigates automatic emotion recognition using visual, audio, and audio-visual features. The authors built two types of emotion recognition models: an individual model, and interpersonal model, capturing interpersonal interaction activities, both verbal and non-verbal.…”