Multimodal architecture for emotion in robots using deep learning

Bansal, Arvind K.; Ghayoumi, Mehdi

doi:10.1109/ftc.2016.7821710

Cited by 17 publications

(8 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Conversely, the use of various sources of information can make the process more resilient against irregularities from the environment (e.g., a noisy room or excessive source of light in the field of view). Albert Mehrabian stated that 7% of the communication information is transferred by language, 38% by paralanguage, and 55% by facial expressions [48] . Through different informative channels, a social robot could be able to receive additional information needed to make its emotion recognition process more sensitive to hidden states of mind.…”

Section: Sensing Modalitiesmentioning

confidence: 99%

Emotion Modelling for Social Robotics Applications: A Review

et al. 2018

View full text Add to dashboard Cite

Robots of today are eager to leave constrained industrial environments and embrace unexplored and unstructured areas, for extensive applications in the real world as service and social robots. Hence, in addition to these new physical frontiers, they must face human ones, too. This implies the need to consider a human-robot interaction from the beginning of the design; the possibility for a robot to recognize users' emotions and, in a certain way, to properly react and "behave". This could play a fundamental role in their integration in society. However, this capability is still far from being achieved. Over the past decade, several attempts to implement automata for different applications, outside of the industry, have been pursued. But very few applications have tried to consider the emotional state of users in the behavioural model of the robot, since it raises questions such as: how should human emotions be modelled for a correct representation of their state of mind? Which sensing modalities and which classification methods could be the most feasible to obtain this desired knowledge? Furthermore, which applications are the most suitable for the robot to have such sensitivity? In this context, this paper aims to provide a general overview of recent attempts to enable robots to recognize human emotions and interact properly.

show abstract

Section: Sensing Modalitiesmentioning

confidence: 99%

Emotion Modelling for Social Robotics Applications: A Review

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Therefore, many studies have effectively used CNN in facial expression recognition to extract features. To enhance the capabilities of HRI and robot-robot interaction, one study [38] proposed a CNN architecture that gives robots the ability to recognize emotions. The network has three convolution layers and one fully connected layer as the output, and information from speech, gestures, and facial recognition is employed as the CNN input.…”

Section: B Deep Neural Networkmentioning

confidence: 99%

CNN and LSTM Based Facial Expression Analysis Model for a Humanoid Robot

Kuo

Tsai

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Robots must be able to recognize human emotions to improve the human-robot interaction (HRI). This study proposes an emotion recognition system for a humanoid robot. The robot is equipped with a camera to capture users' facial images, and it uses this system to recognize users' emotions and responds appropriately. The emotion recognition system, based on a deep neural network, learns six basic emotions: happiness, anger, disgust, fear, sadness, and surprise. First, a convolutional neural network (CNN) is used to extract visual features by learning on a large number of static images. Second, a long short-term memory (LSTM) recurrent neural network is used to determine the relationship between the transformation of facial expressions in image sequences and the six basic emotions. Third, CNN and LSTM are combined to exploit their advantages in the proposed model. Finally, the performance of the emotion recognition system is improved by using transfer learning, that is, by transferring knowledge of related but different problems. The performance of the proposed system is verified through leave-one-out cross-validation and compared with that of other models. The system is applied to a humanoid robot to demonstrate its practicability for improving the HRI. INDEX TERMS Convolutional neural network, long short-term memory, transfer learning, facial expression analysis.

show abstract

“…Deep learning allows computational models to be composed of multiple processing layers for learning the representations of data with multiple levels of abstraction [11]. Deep learning has been used in object detection [12]- [15], emotion recognition [16], [17], semantic segmentation [18], medical diagnosis [19], and many other domains [20]- [23]. The development of deep learning methods has been steadily widening to include multimodal domains [2].…”

Section: Introductionmentioning

confidence: 99%

Multistage Fusion With Dissimilarity Regularization for SAR/IR Target Recognition

et al. 2019

View full text Add to dashboard Cite

In this paper, we propose dissimilarity regularization with a multistage fusion stream for a synthetic aperture radar (SAR) and infrared (IR) sensor fusion using deep learning. The multistage fusion structures are composed of multiple layers for fusing all the feature maps generated by the convolutional neural networks. The proposed structure combines feature maps of equivalent levels, ensuring that the spatial information of the corresponding levels can be utilized for fusion. Dissimilarity regularization is the sum of the normalized cross-correlation between the features generated in two different single-sensor streams. The proposed regularization is added to the conventional learning problem of a single-sensor stream, and each single-sensor stream is promoted to learn the disparate types of features for fusion. To evaluate the proposed algorithm, we compare the recognition rate of the proposed algorithm with that of the conventional fusion approaches using the SAR and IR image databases. Finally, the effects of the proposed architecture and regularization on the fusion result are analyzed.

show abstract

Multimodal architecture for emotion in robots using deep learning

Cited by 17 publications

References 21 publications

Emotion Modelling for Social Robotics Applications: A Review

Emotion Modelling for Social Robotics Applications: A Review

CNN and LSTM Based Facial Expression Analysis Model for a Humanoid Robot

Multistage Fusion With Dissimilarity Regularization for SAR/IR Target Recognition

Contact Info

Product

Resources

About