A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition

Jiang, Yingying; Li, Wei; Hossain, M. Shamim; Chen, Min; Alelaiwi, Abdulhameed; Al-Hammadi, Muneer

doi:10.1016/j.inffus.2019.06.019

Cited by 136 publications

(53 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The current research on emotion recognition also focuses on multimodal fusion which is the combination of sensor data from various modalities to increase the accuracy of the classification of emotions [14,21,36]. There are basically three types of multimodal fusion methods which are data-level, feature-level or early fusion, and decision-level fusion or late fusion [14].…”

Section: Related Workmentioning

confidence: 99%

Emotion Recognition using EEG and Physiological Data for Robot-Assisted Rehabilitation Systems

Gumuslu

Barkana

Köse

2020

Companion Publication of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Emotion Recognition using EEG and Physiological Data for Robot-Assisted Rehabilitation Systems

Gumuslu

Barkana

Köse

2020

Companion Publication of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

“…FER refers to the use of computers to analyze human facial expressions and judge human psychology and emotions through pattern recognition and machine learning algorithms, thereby achieving intelligent human-computer interaction [1]. Traditional FER methods generally include three steps: face detection, feature extraction, and expression recognition [2,3].…”

Section: Introductionmentioning

confidence: 99%

Facial Expression Recognition Algorithm Based on Fusion of Transformed Multilevel Features and Improved Weighted Voting SVM

Meng

Yuan

et al. 2021

Mathematical Problems in Engineering

View full text Add to dashboard Cite

In allusion to the shortcomings of traditional facial expression recognition (FER) that only uses a single feature and the recognition rate is not high, a FER method based on fusion of transformed multilevel features and improved weighted voting SVM (FTMS) is proposed. The algorithm combines the transformed traditional shallow features and convolutional neural network (CNN) deep semantic features and uses an improved weighted voting method to make a comprehensive decision on the results of the four trained SVM classifiers to obtain the final recognition result. The shallow features include local Gabor features, LBP features, and joint geometric features designed in this study, which are composed of distance and deformation characteristics. The deep feature of CNN is the multilayer feature fusion of CNN proposed in this study. This study also proposes to use a better performance SVM classifier with CNN to replace Softmax since the poor distinction between facial expressions. Experiments on the FERPlus database show that the recognition rate of this method is 17.2% higher than that of the traditional CNN, which proves the effectiveness of the fusion of the multilayer convolutional layer features and SVM. FTMS-based facial expression recognition experiments are carried out on the JAFFE and CK+ datasets. Experimental results show that, compared with the single feature, the proposed algorithm has higher recognition rate and robustness and makes full use of the advantages and characteristics of different features.

show abstract

“…The first and easiest involves training a single model for every single source and finally perform a score-level fusion [24]. The second and the hardest requires featurelevel fusion in order to allow the models to take advantage of the intrinsic correlations among different sources [25] but this is typically more difficult as their representations are not always directly compatible. So the problem becomes to find a proper representation of the set of sources to exploit the information presented.…”

Section: Introductionmentioning

confidence: 99%

Affective Robot Story-Telling Human-Robot Interaction: Exploratory Real-Time Emotion Estimation Analysis Using Facial Expressions and Physiological Signals

et al. 2020

View full text Add to dashboard Cite

Affective human-robot interaction is still an active area of research in part due to the great advances in artificial intelligence. Now, the design of autonomous devices that work in real therapeutic environments has become a plausible reality. Affective human-robot interaction requires a robot to analyze the emotional state of the human interlocutor and interpret emotional responses that can be used, not merely in the interaction but, for example, to provoke desired therapeutic responses. It is, therefore, necessary to broaden experimental techniques into more realistic paradigms, where the capacity of emotion estimation can be completely explored. This exploratory paper proposes a realistic experimental paradigm in which the robot employs a dramatic story to evoke emotions in the users, and tests previously self-designed methodologies to be able to make estimates of the users' emotional state in real-time. Regardless of the multiple impediments and restrictions, and all the aspects that could still be improved, this paper can outline the feasibility of the proposed methodology in realistic scenarios.

show abstract

A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition

Cited by 136 publications

References 61 publications

Emotion Recognition using EEG and Physiological Data for Robot-Assisted Rehabilitation Systems

Emotion Recognition using EEG and Physiological Data for Robot-Assisted Rehabilitation Systems

Facial Expression Recognition Algorithm Based on Fusion of Transformed Multilevel Features and Improved Weighted Voting SVM

Affective Robot Story-Telling Human-Robot Interaction: Exploratory Real-Time Emotion Estimation Analysis Using Facial Expressions and Physiological Signals

Contact Info

Product

Resources

About