Towards Formal Multimodal Analysis of Emotions for Affective Computing

Ghayoumi, Mehdi; Bansal, Arvind K.; Thafar, Maha A.

doi:10.18293/dms2016-030

Cited by 14 publications

(6 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several types of modalities were observed in the survey, with some of them being used in several combinations with other input types [50,66,80,83,[208][209][210][211][212][213][214][215][216][217][218][219][220][221][222][223][224]. Facial expression images, being the prime modality in affect identification, were found in most DBs.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

A Survey on Databases for Multimodal Emotion Recognition and an Introduction to the VIRI (Visible and InfraRed Image) Database

Siddiqui

Dhakal

Yang

et al. 2022

MTI

View full text Add to dashboard Cite

Multimodal human–computer interaction (HCI) systems pledge a more human–human-like interaction between machines and humans. Their prowess in emanating an unambiguous information exchange between the two makes these systems more reliable, efficient, less error prone, and capable of solving complex tasks. Emotion recognition is a realm of HCI that follows multimodality to achieve accurate and natural results. The prodigious use of affective identification in e-learning, marketing, security, health sciences, etc., has increased demand for high-precision emotion recognition systems. Machine learning (ML) is getting its feet wet to ameliorate the process by tweaking the architectures or wielding high-quality databases (DB). This paper presents a survey of such DBs that are being used to develop multimodal emotion recognition (MER) systems. The survey illustrates the DBs that contain multi-channel data, such as facial expressions, speech, physiological signals, body movements, gestures, and lexical features. Few unimodal DBs are also discussed that work in conjunction with other DBs for affect recognition. Further, VIRI, a new DB of visible and infrared (IR) images of subjects expressing five emotions in an uncontrolled, real-world environment, is presented. A rationale for the superiority of the presented corpus over the existing ones is instituted.

show abstract

Section: Conclusion and Discussionmentioning

confidence: 99%

A Survey on Databases for Multimodal Emotion Recognition and an Introduction to the VIRI (Visible and InfraRed Image) Database

Siddiqui

Dhakal

Yang

et al. 2022

MTI

View full text Add to dashboard Cite

show abstract

“…Choosing the most probable result as the final result, the robot responded to different emotions expressed by the human. Jonathan et al [30] proposed a robust multimodal interaction framework that can realize the interaction between rescuers and flying robots by integrating voice and hand and arm posture data based on post fusion integration in the decision-making layer.…”

Section: A Virtual Teachingmentioning

confidence: 99%

A Design of Smart Beaker Structure and Interaction Paradigm Based on Multimodal Fusion Understanding

et al. 2020

View full text Add to dashboard Cite

Virtual chemistry experiment is an important teaching tool in middle schools. It can not only help students understand the experimental principles, but also help them memorize the experimental procedures. However, there are still some problems in virtual chemistry experiments. First, in existing research, user intentions are often misunderstood and cannot be accurately understood. Second, the existing research fail to identify the user's wrong actions, which reduce the accuracy of the experiment. Third, the user sense of operation and realism is not strong during the experiment, which reduces the user's experience. Finally, the lack of navigation guidance for experimental operations increases user learning time. In order to solve these problems. This paper proposes a scheme for establishing an intelligent navigational chemical laboratory based on multimodal fusion. First of all, we design a new smart beaker structure with perceptual ability, which can be used to complete most chemical experiments and give users a real sense of experience. Besides, we propose a multimodal fusion understanding algorithm, which reduces the misidentification of the experiment and better understands the real intention of the user. Finally, intelligent navigation and wrong behavior recognition functions are added to the experimental equipment, which improves the efficiency of human-computer interaction. The results show that Compared with the existing virtual laboratory or system, the chemical laboratory scheme proposed in this paper through multimodal fusion understanding algorithm greatly reduces the user's memory load and improves the success rate of the experiment. Moreover, through the combination of virtual and real, the virtual chemistry experiment not only improves the authenticity of the operation, but also stimulates the students' interest in learning, which is well received by users.

show abstract

“…The FACS is based on the simulation of facial muscle movement. An action unit (AU) is comprised of segments of the muscles involved in facial expression [6]. Seventeen major AUs are involved in basic facial expression, and all facial expressions are determined by the FACS through identification of these AUs.…”

Section: Introductionmentioning

confidence: 99%

CNN and LSTM Based Facial Expression Analysis Model for a Humanoid Robot

Kuo

Tsai

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Robots must be able to recognize human emotions to improve the human-robot interaction (HRI). This study proposes an emotion recognition system for a humanoid robot. The robot is equipped with a camera to capture users' facial images, and it uses this system to recognize users' emotions and responds appropriately. The emotion recognition system, based on a deep neural network, learns six basic emotions: happiness, anger, disgust, fear, sadness, and surprise. First, a convolutional neural network (CNN) is used to extract visual features by learning on a large number of static images. Second, a long short-term memory (LSTM) recurrent neural network is used to determine the relationship between the transformation of facial expressions in image sequences and the six basic emotions. Third, CNN and LSTM are combined to exploit their advantages in the proposed model. Finally, the performance of the emotion recognition system is improved by using transfer learning, that is, by transferring knowledge of related but different problems. The performance of the proposed system is verified through leave-one-out cross-validation and compared with that of other models. The system is applied to a humanoid robot to demonstrate its practicability for improving the HRI. INDEX TERMS Convolutional neural network, long short-term memory, transfer learning, facial expression analysis.

show abstract

Towards Formal Multimodal Analysis of Emotions for Affective Computing

Cited by 14 publications

References 15 publications

A Survey on Databases for Multimodal Emotion Recognition and an Introduction to the VIRI (Visible and InfraRed Image) Database

A Survey on Databases for Multimodal Emotion Recognition and an Introduction to the VIRI (Visible and InfraRed Image) Database

A Design of Smart Beaker Structure and Interaction Paradigm Based on Multimodal Fusion Understanding

CNN and LSTM Based Facial Expression Analysis Model for a Humanoid Robot

Contact Info

Product

Resources

About